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(54) Method and apparatus for panoramic imaging, control method therefor, and storage medium 



(57) This invention has as its object to provide a 
convenient image processing apparatus which does not 
limit image sensing purposes. To achieve this object, an 
image processing apparatus has a recording medium 
for recording and playing back an image sensed by an 
image sensing unit, and an image display unit for play- 
ing back and displaying at least an image sensed before 



the current image. The image display unit has a display 
layout mode for displaying the current image sensing 
signal and at least the image sensed before the current 
image sensing signal with their partial boundary regions 
overlapping each other. 
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Description 

BACKGROUND OF THE INVENTION 

[0001] The present invention relates to an image 
processing method and apparatus for sensing, record- 
ing, and playing back stfll and moving images, a control 
method therefor, and a storage medium which stores a 
control program for controlling the image processing 
apparatus. 

[0002] Conventionally, image processing apparatuses 
such as electronic cameras and the like, which 
record/play back still and moving images using memory 
cards having solid state memory elements as recording 
media have already been commercially available, and 
electronic cameras with electronic viewfinders such as 
color liquid crystal panels and the like have also been 
commercially available. 

[0003] These electronic cameras allow the electronic 
camera user to determine the camera angle by continu- 
ously displaying images before sensing, and to confirm 
the sensed images by playing back and displaying 
them. 

[0004] Especially, the function of generating a pano- 
ramic image by combining a plurality of sensed images 
is very convenient and effective for the electronic cam- 
era user. 

[0005] However, in such conventional image process- 
ing apparatus such as an electronic camera or the like, 
since it is hard to sense a panoramic image with a com- 
plicated combination of images or to provide a view- 
finder display which is convenient upon sensing a 
panoramic image with a complicated combination of 
images, its image sensing purpose is limited. 
[0006] When the user of the image processing appa- 
ratus such as an electronic camera or the like has deter- 
mined the camera angle, since he or she cannot start 
panoramic image sensing in an arbitrary direction, a 
convenient image processing apparatus cannot be pro- 
vided. 

[0007] Furthermore, since neither viewfinder display 
nor operation that facilitate re-sensing are available 
immediately after sensing of a panoramic image, a con- 
venient image processing apparatus cannot be pro- 
vided. 

[0008] Electronic cameras having variable focus 
lenses or zoom lenses that can vary the focal length of 
a photographing lens are also commercially available. 
However, in such conventional image processing appa- 
ratus such as an electronic camera or the like, the user 
of the image processing apparatus such as an elec- 
tronic camera or the like may switch the focal length of a 
photographing lens or zoom during sensing a pano- 
ramic image. In such case, upon generating a pano- 
ramic image by combining a plurality of sensed images, 
extra computations for correcting different focal lengths 
are needed, resulting in a considerable drop of image 
quality. 



[0009] When individual sensed images have consid- 
erably different angles due to switching of the focal 
length of the photographing lens or zooming, the outer 
shape of a synthesized panoramic image varies in units 

5 of sensed images and has staircasing. 

[0010] Furthermore, when the user notices during 
panoramic image sensing that the focal length of the 
photographing lens has been switched to a side which is 
not suitable for panoramic image sensing, or that the 

io zoom position of the zoom lens is not suitable for pano- 
ramic image sensing, he or she may switch the focal 
length of the photographing lens or zoom even while a 
plurality of images are being sensed for panoramic 
image sensing, thus posing the above-mentioned prob- 

15 lems. 

[001 1 ] In an electronic camera that uses exchangea- 
ble lenses, it is possible for the electronic camera user 
to determine the camera angle of a panoramic image 
and sense in correspondence with the selected lens, 

20 and such function is effective for the electronic camera 
user. However, in such conventional image processing 
apparatus such as an electronic camera or the like, the 
user of the image processing apparatus such as an 
electronic camera or the like may exchange the photo- 

25 graphing lens by another lens with a different focal 
length. In such case, upon generating a panoramic 
image by combining a plurality of sensed images, extra 
computations for correcting different focal lengths 
between the former and latter lenses are needed, 

30 resulting in a considerable drop of image quality. 

[001 2] When individual sensed images have consid- 
erably different angles due to exchange of the photo- 
graphing lens by another lens with a different focal 
length, the outer shape of a synthesized panoramic 

35 image varies in units of sensed images and has stair- 
casing. 

[0013] Not only upon panoramic image sensing but 
also upon mounting a detached photographing lens 
again, the previous setup state corresponding to that 
40 photographing lens must be re-set. resulting in incon- 
venience. 

SUMMARY OF THE INVENTION 

45 [0014] The present invention has been made in con- 
sideration of the aforementioned problems of the prior 
art and has as its object to provide a convenient image 
processing method and apparatus, which do not limit 
image sensing purposes. 

so [001 5] It is another object of the present invention to 
provide a storage medium storing a control program 
that can smoothly control the image processing appara- 
tus of the present invention. 

[001 6] It is still another object of the present invention 
55 to provide an image processing apparatus which does 
not pose any problem in panoramic image sensing even 
when a lens having a function of changing its focal 
length is used, its control method, and a storage 
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medium. 

[001 7] tt is still another object of the present invention 
to provide an image processing apparatus which does 
not pose any problem in panoramic image sensing even 
when a lens is exchangeable, its control method, and a 
storage medium. 

[0018] In order to solve the above-mentioned prob- 
lems and to achieve the above objects, an image 
processing method according to the present invention is 
characterized by the following arrangement according 
to its first aspect. 

[001 9] That is, there is provided an image processing 
method for recording a plurality of sensed images on a 
recording medium, and playing back and displaying the 
images, comprising: the image sensing step of sensing 
an image; the recording/playback step of recording and 
playing back the image sensed in the image sensing 
step; and the display step of playing back and displaying 
an image sensed at least before a current image, 
wherein the display step comprises a plurality of display 
layout modes for displaying the current image sensing 
signal, and a signal obtained by playing back the image 
sensed at least before the current image with partial 
boundary regions thereof overlapping each other. 
[0020] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its first aspect. 
[0021] That is, there is provided an image processing 
apparatus for recording a plurality of sensed images on 
a recording medium, and playing back and displaying 
the images, comprising: image sensing means for sens- 
ing an image; recording/playback means for recording 
and playing back the image sensed by the image sens- 
ing means; and display means for playing back and dis- 
playing an image sensed at least before a current 
image, wherein the display means comprises a plurality 
of display layout modes for displaying the current image 
sensing signal, and a signal obtained by playing back 
the image sensed at least before the current image with 
partial boundary regions thereof overlapping each 
other. 

[0022] An image processing method according to the 
present invention is characterized by the following 
arrangement according to its second aspect. 
[0023] That is, there is provided an image processing 
method for recording a plurality of sensed images on a 
recording medium, and playing back and displaying the 
images, comprising: the image sensing step of sensing 
an image; the recording/playback step of recording and 
playing back the image sensed in the image sensing 
step; and the display step of playing back and displaying 
an image sensed at least before a current image, 
wherein the display step comprises a display layout 
mode for laying out and displaying the current image 
sensing signal, and a signal obtained by playing back 
the image sensed at least before the current image in 
one direction with partial boundary regions thereof over- 
lapping each other, and a function of reversing the lay- 



out direction in the one direction. 
[0024] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its second aspect. 

5 [0025] That is. there is provided an image processing 
apparatus for recording a plurality of sensed images on 
a recording medium, and playing back and displaying 
the images, comprising: image sensing means for sens- 
ing an image; recording/playback means for recording 

w and playing back the image sensed by the image sens- 
ing means; and display means for playing back and dis- 
playing an image sensed at least before a current 
image, wherein the display means comprises a display 
layout mode for laying out and displaying the current 

75 image sensing signal, and a signal obtained by playing 
back the image sensed at least before the current image 
in one direction with partial boundary regions thereof 
overlapping each other, and a function of reversing the 
layout direction in the one direction. 

20 [0026] An image processing method according to the 
present invention is characterized by the following 
arrangement according to its third aspect. 
[0027] That is, there is provided an image processing 
method for recording a plurality of sensed images on a 

25 recording medium, and playing back and displaying the 
images, comprising: the image sensing step of sensing 
an image; the recording/playback step of recording and 
playing back the image sensed in the image sensing 
step; the display step of playing back and displaying an 

30 image sensed at least before a current image; and the 
switching step of switching a first display layout mode for 
laying out and displaying the current image sensing sig- 
nal, and a signal obtained by playing back the image 
sensed at least before the current image in one direc- 

35 tion with partial boundary regions thereof overlapping 
each other, and a second display layout mode for laying 
out and displaying the signal obtained by playing back 
the image sensed at least before the current image, and 
a signal obtained by playing back an image sensed still 

40 before the signal in the one direction with partial bound- 
ary regions thereof overlapping each other. 
[0028] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its third aspect. 

45 [0029] That is, there is provided an image processing 
apparatus for recording a plurality of sensed images on 
a recording medium, and playing back and displaying 
the images, comprising: image sensing means for sens- 
ing an image; recording/playback means for recording 

so and playing back the image sensed by the image sens- 
ing means; display means for playing back and display- 
ing an image sensed at least before a current image; 
and switching means for switching a first display layout 
mode for laying out and displaying the current image 

55 sensing signal, and a signal obtained by playing back 
the image sensed at least before the current image in 
one direction with partial boundary regions thereof over- 
lapping each other, and a second display layout mode 
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for laying out and displaying the signal obtained by play- 
ing back the image sensed at least before the current 
image, and a signal obtained by playing back an image 
sensed still before the signal in the one direction with 
partial boundary regions thereof overlapping each 
other. 

[0030] An image processing method according to the 
present invention is characterized by the following 
arrangement according to its fourth aspect. 
[0031] That is, there is provided an image processing 
method for recording a plurality of sensed images on a 
recording medium, and playing back and displaying the 
images, comprising: the image sensing step of sensing 
an image; the recording/playback step of recording and 
playing back the image sensed in the image sensing 
step; and the display step of playing back and displaying 
an image sensed at least before a current image, 
wherein the display step comprises a display layout 
mode for laying out and displaying the current image 
sensing signal, and a signal obtained by playing back 
the image sensed at least before the current image in 
two directions with partial boundary regions thereof 
overlapping each other, and includes the selection step 
of selecting an arbitrary one of display regions laid out in 
the display layout mode. 

[0032] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its fourth aspect. 
[0033] That is, there is provided an image processing 
apparatus for recording a plurality of sensed images on 
a recording medium, and playing back and displaying 
the images, comprising: image sensing means for sens- 
ing an image; recording/playback means for recording 
and playing back the image sensed by the image sens- 
ing means; and display means for playing back and dis- 
playing an image sensed at least before a current 
image, wherein the display means comprises a display 
layout mode for laying out and displaying the current 
image sensing signal, and a signal obtained by playing 
back the image sensed at least before the current image 
in two directions with partial boundary regions thereof 
overlapping each other, and includes selection means 
for selecting an arbitrary one of cfi splay regions laid out 
in the display layout mode. 

[0034] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its first aspect. 

[0035] That is, there is provided a storage medium 
which stores a control program for controlling an image 
processing apparatus for recording a plurality of sensed 
images on a recording medium, and playing back and 
displaying the images, the control program having con- 
trol modules of the steps of: sensing an image; record- 
ing and playing back the sensed image; playing back 
and displaying an image sensed at least before a cur- 
rent image; and controlling to execute a plurality of dis- 
play layout modes for displaying the current image 
sensing signal, and a signal obtained by playing back 



the image sensed at least before the current image with 
partial boundary regions thereof overlapping each 
other. 

[0036] A storage medium according to the present 
5 invention is characterized by the following arrangement 
according to its second aspect. 
[0037] That is, there is provided a storage medium 
which stores a control program for controlling an image 
processing apparatus for recording a plurality of sensed 
io images on a recording medium, and playing back and 
displaying the images, the control program having con- 
trol modules of the steps of: sensing an image; record- 
ing and playing back the sensed image; playing back 
and displaying an image sensed at least before a cur- 
is rent image; and controlling to execute a display layout 
mode for laying out and displaying the current image 
sensing signal, and a signal obtained by playing back 
the image sensed at least before the current image in 
one direction with partial boundary regions thereof over- 
do lapping each other, and a function of reversing the lay- 
out direction in the one direction. 
[0038] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its third aspect. 
25 [0039] That is, there is provided a storage medium 
which stores a control program for controlling an image 
processing apparatus for recording a plurality of sensed 
images on a recording medium, and playing back and 
displaying the images, the control program having con- 
30 trol modules of the steps of: sensing an image; record- 
ing and playing back the sensed image; playing back 
and displaying an image sensed at least before a cur- 
rent image; and controlling to switch a first display layout 
mode for laying out and displaying the current image 
35 sensing signal, and a signal obtained by playing back 
the image sensed at least before the current image in 
one direction with partial boundary regions thereof over- 
lapping each other, and a second display layout mode 
for laying out and displaying the signal obtained by play- 
40 ing back the image sensed at least before the current 
image, and a signal obtained by playing back an image 
sensed still before the signal in the one direction with 
partial boundary regions thereof overlapping each 
other. 

45 [0040] A storage medium according to the present 
invention is characterized by the following anangement 
according to its fourth aspect. 

[0041] That is. there is provided a storage medium 
which stores a Control program for controlling an image 

so processing apparatus for recording a plurality of sensed 
images on a recording medium, and playing back and 
displaying the images, the control program having con- 
trol modules of the steps of: sensing an image; record- 
ing and playing back the sensed image: playing back 

55 and displaying an image sensed at least before a cur- 
rent image; and controlling to select an arbitrary one of 
display regions laid out in a display layout mode for lay- 
ing out and displaying the current image sensing signal, 
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and a signal obtained by playing back the image sensed 
at least before the current image in two directions with 
partial boundary regions thereof overlapping each 
other. 

[0042] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its fifth aspect. 
[0043] That is, there is provided an image processing 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 
means, comprising: image sensing means comprising 
an image sensing lens which can change a focal length; 
storage means for storing a plurality of images sensed 
by the image sensing means in association with each 
other; focal length change instruction means for output- 
ting an instruction for changing the focal length of the 
image sensing lens; and control means for controlling to 
inhibit the focal length of the image sensing lens from 
changing upon reception of the instruction for changing 
the focal length of the image sensing lens from the focal 
length change instruction means after a first one of the 
plurality of images to be stored in the storage means in 
association with each other is sensed and stored. 
[0044] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its sixth aspect. 
[0045] That is. there is provided an image processing 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 
means, comprising: image sensing means comprising 
an image sensing lens which can change a focal length; 
storage means for storing a plurality of images sensed 
by the image sensing means in association with each 
other; focal length change instruction means for output- 
ting an instruction for changing the focal length of the 
image sensing lens; display means for displaying prede- 
termined information on an external unit; and control 
means for controlling the display means to display alert 
information upon reception of the instruction for chang- 
ing the focal length of the image sensing lens from the 
focal length change instruction means after a first one of 
the plurality of images to be stored in the storage means 
in association with each other is sensed and stored. 
[0046] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its seventh aspect. 
[0047] That is, there is provided an image processing 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 
means, comprising: image sensing means comprising 
an image sensing lens which can change a focal length; 
storage means for storing a plurality of images sensed 
by the image sensing means in association with each 
other; focal length change instruction means for output- 
ting an instruction for changing the focal length of the 
image sensing lens; display means for displaying prede- 
termined information on an external unit; and control 
_ means for controlling the display means to display alert 



information and inhibiting the focal length of the image 
sensing lens from changing upon reception of the 
instruction for changing the focal length of the image 
sensing lens from the focal length change instruction 
5 means after a f irst one of the plurality of images to be 
stored in the storage means in association with each 
other is sensed and stored. 

[0048] An image processing apparatus according to 
the present invention is characterized by the following 

io arrangement according to its eighth aspect. 

[0049] That is. there is provided an image processing 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 
means, comprising: image sensing means comprising 

75 an image sensing lens which can change a focal length; 
storage means for storing a plurality of images sensed 
by, the image sensing means in association with each 
other; focal length change instruction means for output- 
ting an instruction for changing the focal length of the 

20 image sensing lens; and control means for controlling to 
start image sensing of a plurality of new images to be 
stored in association with each other upon reception of 
the instruction for changing the focal length of the image 
sensing lens from the focal length change instruction 

25 means after a first one of the plurality of images to be 
stored in the storage means in association with each 
other is sensed and stored. 

[0050] An image processing apparatus according to 
the present invention is characterized by the following 

30 arrangement according to its ninth aspect. 

[0051 ] That is. there is provided an image processing 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 
means, comprising: image sensing means comprising 

35 an image sensing lens which can change a focal length; 
storage means for storing a plurality of images sensed 
by the image sensing means in association with each 
other; focal length change instruction means for output- 
ting an instruction tor changing the focal length of the 

40 image sensing lens; display means for displaying prede- 
termined information on an external unit; and control 
means for controlling the display means to display alert 
information and then starting image sensing of a plural- 
ity of new images to be stored in association with each 

45 other upon reception of the instruction for changing the 
focal length of the image sensing lens from the focal 
length change instruction means after a first one of the 
plurality of images to be stored in the storage means in 
association with each other is sensed and stored. 

so [0052] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its tenth aspect. 
[0053] That is, there is provided an image processing 
apparatus having a function of storing a plurality of 

55 sensed still images and/or moving images in storage 
means, comprising: image sensing means comprising 
an image sensing lens which can change a focal length; 
storage means for storing a plurality of images sensed 
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by the image sensing means in association with each 
other; focal length change instruction means for output- 
ting an instruction for changing the focal length of the 
image sensing lens; and control means for controlling to 
set the focal length of the image sensing lens at an ini- 
tial value before sensing of a first one of the plurality of 
images to be stored in the storage means in association 
with each other is started. 

[0054] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
first aspect. 

[0055] That is, there is provided a control method for 
an image processing apparatus which comprises image 
sensing means comprising an image sensing lens 
- — which can change a focal length, storage means for 
storing a plurality of images sensed by the image sens- 
ing means in association with each other, and focal 
length change instruction means for outputting an 
instruction for changing the focal length of the image 
sensing lens, comprising the step of: inhibiting the focal 
length of the image sensing lens from changing upon 
reception of the instruction for changing the focal length 
of the image sensing lens from the focal length change 
instruction means after a first one of the plurality of 
images to be stored in the storage means in association 
with each other is sensed and stored. 
[0056] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
second aspect. 

[0057] That is, there is provided a control method for 
an image processing apparatus which comprises image 
sensing means comprising an image sensing lens 
which can change a focal length, storage means for 
storing a plurality of images sensed by the image sens- 
ing means in association with each other, focal length 
change instruction means for outputting an instruction 
for changing the focal length of the image sensing lens, 
and display means for displaying predetermined infor- 
mation on an external unit, comprising the step of: con- 
trolling the display means to display alert information 
upon reception of the instruction for changing the focal 
length of the image sensing lens from the focal length 
change instruction means after a first one of the plurality 
of images to be stored in the storage means in associa- 
tion with each other is sensed and stored. 
[0058] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
third aspect. 

[0059] That is, there is provided a control method for 
an image processing apparatus which comprises image 
sensing means comprising an image sensing lens 
which can change a focal length, storage means for 
storing a plurality of images sensed by the image sens- 
ing means in association with each other, focal length 
change instruction means for outputting an instruction 



for changing the focal length of the image sensing lens, 
and display means for displaying predetermined infor- 
mation on an external unit, comprising the step of: con- 
trolling the display means to display alert information 

5 and inhibiting the focal length of the image sensing lens 
from changing upon reception of the instruction for 
changing the focal length of the image sensing lens 
from the focal length change instruction means after a 
first one of the plurality of images to be stored in the 

10 storage means in association with each other is sensed 
and stored. 

[0060] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 

is fourth aspect. 

[0061] That is. there is provided a control method for 
an image processing apparatus which comprises image 
sensing means comprising an image sensing lens 
which can change a focal length, storage means for 

20 storing a plurality of images sensed by the image sens- 
ing means in association with each other, and focal 
length change instruction means for outputting an 
instruction for changing the focal length of the image 
sensing lens, comprising the step of: starting image 

25 sensing of a plurality of new images to be stored in 
association with each other upon reception of the 
instruction for changing the focal length of the image 
sensing lens from the focal length change instruction 
means after a first one of the plurality of images to be 

30 stored in the storage means in association with each 
other is sensed and stored. 

[0062] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
35 fifth aspect 

[0063] That is, there is provided a control method for 
an image processing apparatus which comprises image 
sensing means comprising an image sensing lens 
which can change a focal length, storage means for 
40 storing a plurality of images sensed by the image sens- 
ing means in association with each other, focal length 
change instruction means for outputting an instruction 
for changing the focal length of the image sensing lens, 
and display means for displaying predetermined infor- 
ms mation on an external unit, comprising the step of: con- 
trolling the display means to display alert information 
and then starting image sensing of a plurality of new 
images to be stored in association with each other upon 
reception of the instruction for changing the focal length 
so of the image sensing lens from the focal length change 
instruction means after a first one of the plurality of 
images to be stored in the storage means in association 
with each other is sensed and stored. 
[0064] A control method for an image processing 
55 apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
sixth aspect. 

[0065] That is, there is provided a control method for 
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an image processing apparatus which comprises image 
sensing means comprising an image sensing lens 
which can change a focal length, storage means for 
storing a plurality of images sensed by the image sens- 
ing means in association with each other, and focal 
length change instruction means for outputting an 
instruction for changing the focal length of the image 
sensing lens, comprising the step of: setting the focal 
length of the image sensing lens at an initial value 
before sensing of a first one of the plurality of images to 
be stored in the storage means in association with each 
other is started. 

[0066] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its fifth aspect. 

[0067] That is, there is provided a storage medium 
that stores a control program for controlling an image 
processing apparatus which comprises image sensing 
means comprising an image sensing lens which can 
change a focal length, storage means for storing a plu- 
rality of images sensed by the image sensing means in 
association with each other, and focal length change 
instruction means for outputting an instruction for 
changing the focal length of the image sensing lens, the 
control program comprising a code of the step of: inhib- 
iting the focal length of the image sensing lens from 
changing upon reception of the instruction for changing 
the focal length of the image sensing lens from the focal 
length change instruction means after a first one of the 
plurality of images to be stored in the storage means in 
association with each other is sensed and stored. 
[0068] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its sixth aspect. 

[0069] That is, there is provided a storage medium 
that stores a control program for controlling an image 
processing apparatus which comprises image sensing 
means comprising an image sensing lens which can 
change a focal length, storage means for storing a plu- 
rality of images sensed by the image sensing means in 
association with each other, focal length change instruc- 
tion means for outputting an instruction for changing the 
focal length of the image sensing lens, and display 
means for displaying predetermined information on an 
external unit, the control program comprising a code of 
the step of: controlling the display means to display alert 
information upon reception of the instruction for chang- 
ing the focal length of the image sensing lens from the 
focal length change instruction means after a first one of 
the plurality of images to be stored in the storage means 
in association with each other is sensed and stored. 
[0070] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its seventh aspect. 
[0071] That is. there is provided a storage medium 
that stores a control program for controlling an image 
processing apparatus which comprises image sensing 
means comprising an image sensing lens which can 



change a focal length, storage means for storing a plu- 
rality of images sensed by the image sensing means in 
association with each other, focal length change instruc- 
tion means for outputting an instruction for changing the 

5 focal length of the image sensing lens, and display 
means for displaying predetermined information on an 
external unit, the control program comprising a code of 
the step of: controlling the display means to display alert 
information and inhibiting the focal length of the image 

70 sensing lens from changing upon reception of the 
instruction for changing the focal length of the image 
sensing lens from the focal length change instruction 
means after a first one of the plurality of images to be 
stored in the storage means in association with each 

is other is sensed and stored. 

[0072] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its eighth aspect. 

[0073] That is, there is provided a storage medium 

20 that stores a control program for controlling an image 
processing apparatus which comprises image sensing 
means comprising an image sensing lens which can 
change a focal length, storage means for storing a plu- 
rality of images sensed by the image sensing means in 

25 association with each other, and focal length change 
instruction means for outputting an instruction for 
changing the focal length of the image sensing lens, the 
control program comprising a code of the step of: start- 
ing image sensing of a plurality of new images to be 

30 stored in association with each other upon reception of 
the instruction for changing the focal length of the image 
sensing lens from the focal length change instruction 
means after a first one of the plurality of images to be 
stored in the storage means in association with each 

35 other is sensed and stored. 

[0074] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its ninth aspect. 

[0075] That is, there is provided a storage medium 

40 that stores a control program for controlling an image 
processing apparatus which comprises image sensing 
means comprising an image sensing lens which can 
change a focal length, storage means for storing a plu- 
rality of images sensed by the image sensing means in 

45 association with each other, focal length change instruc- 
tion means for outputting an instruction for changing the 
focal length of the image sensing lens, and display 
means for displaying predetermined information on an 
external unit.^the control program comprising a code of 

so the step of: controlling the display means to display alert 
information and then starting image sensing of a plural- 
ity of new images to be stored in association with each 
other upon reception of the instruction for changing the 
focal length of the image sensing lens from the focal 

55 length change instruction means after a first one of the 
plurality of images to be stored in the storage means in 
association with each other is sensed and stored. 
[0076] A storage medium according to the present 
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invention is characterized by the following arrangement 
according to its tenth aspect. 

[0077] That is, there is provided a storage medium 
that stores a control program for controlling an image 
processing apparatus which comprises image sensing 
means comprising an image sensing lens which can 
change a focal length, storage means for storing a plu- 
rality of images sensed by the image sensing means in 
association with each other, and focal length change 
instruction means for outputting an instruction for 
changing the focal length of the image sensing lens, the 
control program comprising a code of the step of: set- 
ting the focal length of the image sensing lens at an ini- 
tial value before sensing of a first one of the plurality of 
images to be stored in the storage means in association 
with each other is started. - 
[0078] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its eleventh aspect. 
[0079] That is, there is provided an image processing 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 
means, comprising: a detachable lens unit; image sens- 
ing means for sensing an object image formed by the 
lens unit; storage means for storing a plurality of images 
sensed by the image sensing means in association with 
each other; detection means for detecting attach- 
ment/detachment of the lens unit; display means for dis- 
playing predetermined information on an external unit; 
and control means for controlling the display means to 
display alert information when the detection means 
detects detachment of the lens unit after a first one of 
the plurality of images to be stored in the storage means 
in association with each other is sensed and stored. 
[0080] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its twelfth aspect. 
[0081] That is, there is provided an image processing 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 
means, comprising: a detachable lens unit; image sens- 
ing means for sensing an object image formed by the 
lens unit; storage means for storing a plurality of images 
sensed by the image sensing means in association with 
each other; detection means for detecting attach- 
ment/detachment of the lens unit; and control means for 
controlling to start sensing of a plurality of new images 
to be stored in association with each other when the 
detection means detects detachment of the lens unit 
after a first one of the plurality of images to be stored in 
the storage means in association with each other is 
sensed and stored. 

[0082] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its thirteenth aspect. 
[0083] That is, there is provided an image processing 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 



means, comprising: a detachable lens unit; image sens- 
ing means for sensing an object image formed by the 
lens unit; storage means for storing a plurality of images 
sensed by the image sensing means in association with 

5 each other; detection means for detecting attach- 
ment/detachment of the lens unit; display means for dis- 
playing predetermined information on an external unit; 
and control means for controlling the display means to 
display alert information and then starting sensing of a 

io plurality of new images to be stored in association with 
each other when the detection means detects detach- 
ment of the lens unit after a first one of the plurality of 
images to be stored in the storage means in association 
with each other is sensed and stored. 

15 [0084] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its fourteenth aspect. 
[0085] That is, there is provided an image processing 
apparatus having a function of storing a plurality of 

20 sensed still images and/or moving images in storage 
means, comprising: a detachable lens unit; image sens- 
ing means for sensing an object image formed by the 
lens unit; storage means for storing a plurality of images 
sensed by the image sensing means in association with 

25 each other; instruction means for instructing attach- 
ment/detachment of the lens unit; and control means for 
controlling to inhibit the lens unit from being detached 
when the instruction means outputs an instruction for 
detaching the lens unit after a first one of the plurality of 

30 images to be stored in the storage means in association 
with each other is sensed and stored. 
[0086] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its fifteenth aspect. 

35 [0087] That is, there is provided an image processing 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 
means, comprising: a detachable lens unit; image sens- 
ing means for sensing an object image formed by the 

40 lens unit; storage means for storing a plurality of images 
sensed by the image sensing means in association with 
each other; instruction means for instructing attach- - 
ment/detachment of the lens unit; display means for dis- 
playing predetermined information on an external unit; 

45 and control means for controlling the display means to 
display alert information and inhibiting the lens unit from 
being detached when the instruction means outputs an 
instruction for detaching the lens unit after a first one of 
the plurality of images to be stored in the storage means 

so in association with each other is sensed and stored. 
[0088] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its sixteenth aspect. 
[0089] That is. there is provided an image processing 

55 apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 
means, comprising: a detachable lens unit; image sens- 
ing means for sensing an object image formed by the 
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lens unit; storage means for storing a plurality of images 
sensed by the image sensing means in association with 
each other; instruction means for instructing attach- 
ment/detachment of the lens unit; and control means for 
controlling to permit detachment of the lens unit when 5 
the instruction means outputs an instruction for detach- 
ing the lens unit before a first one of the plurality of 
images to be stored in the storage means in association 
with each other is sensed and stored, and for controlling 
to inhibit the lens unit from being detached when the 10 
instruction means outputs an instruction for detaching 
the lens unit after a first one of the plurality of images to 
be stored in the storage means in association with each 
other is sensed and stored. 

[0090] An image processing apparatus according to 75 
the present invention is characterized by the following 
arrangement according to its seventeenth aspect. 
[0091] That is, there is provided an image processing 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 20 
means, comprising: a detachable lens unit having non- 
volatile storage means; image sensing means for sens- 
ing an object image formed by the lens unit; storage 
means lor storing a plurality of images sensed by the 
image sensing means in association with each other; 25 
instruction means for instructing attachment/detach- 
ment of the lens unit; and control means for controlling 
to permit detachment of the lens unit after information 
pertaining to an operation state of the lens unit is stored 
in the nonvolatile storage means, when the instruction 30 
means outputs an instruction for detaching the lens unit, 
and for reading out information pertaining to a use state 
of the lens unit stored in the nonvolatile storage means 
and re-setting an operation state of the image process- 
ing apparatus in accordance with the readout informa- 35 
tion, when the lens unit is attached again. 
[0092] An image processing apparatus according to 
the present invention is characterized by the following 
arrangement according to its eighteenth aspect. 
[0093] That is. there is provided an image processing 40 
apparatus having a function of storing a plurality of 
sensed still images and/or moving images in storage 
means, comprising: a detachable lens unit having non- 
volatile storage means; image sensing means for sens- 
ing an object image formed by the lens unit; storage 45 
means for storing a plurality of images sensed by the 
image sensing means in association with each other; 
instruction means for instructing attachment/detach- 
ment of the lens unit; and control means for controlling 
to permit detachment of the lens unit after information so 
pertaining to an operation state of the lens unit and user 
information are stored in the nonvolatile storage means, 
when the instruction means outputs an instruction for 
detaching the lens unit, and for reading out information 
pertaining to a use state of the lens unit and the user 55 
information stored in the nonvolatile storage means and 
re-setting an operation state of the image processing 
apparatus in accordance with the readout information in 



unit of users, when the lens unit is attached again. 
[0094] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
seventh aspect. 

[0095] That is. there is provided a control method for 
an image processing apparatus which comprises a 
detachable lens unit, image sensing means for sensing 
an object image formed by the lens unit, storage means 
for storing a plurality of images sensed by the image 
sensing means in association with each other, detection 
means for detecting attachment/detachment of the lens 
unit, and display means for displaying predetermined 
information on an external unit, comprising the step of: 
controlling the display means to display alert informa- 
tion when the detection means detects detachment of 
the lens unit after a first one of the plurality of images to 
be stored in the storage means in association with each 
other is sensed and stored. 

[0096] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
eighth aspect. 

[0097] That is. there is provided a control method for 
an image processing apparatus which comprises a 
detachable lens unit, image sensing means for sensing 
an object image formed by the lens unit, storage means 
for storing a plurality of images sensed by the image 
sensing means in association with each other, and 
detection means for detecting attachment/detachment 
of the lens unit, comprising the step of: starting sensing 
of a plurality of new images to be stored in association 
with each other when the detection means detects 
detachment of the lens unit after a first one of the plural- 
ity of images to be stored in the storage means in asso- 
ciation with each other is sensed and stored. 
[0098] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
ninth aspect. 

[0099] That is. there is provided a control method for 
an image processing apparatus which comprises a 
detachable lens unit, image sensing means for sensing 
an object image formed by the lens unit, storage means 
for storing a plurality of images sensed by the image 
sensing means in association with each other, detection 
means for detecting attachment/detachment of the lens 
unit, and display means for displaying predetermined 
information on an external unit, comprising the step of: 
controlling the display means to display alert informa- 
tion and then starting sensing of a plurality of new 
images to be stored in association with each other when 
the detection means detects detachment of the lens unit 
after a first one of the plurality of images to be stored in 
the storage means in association with each other is 
sensed and stored. 

[0100] A control method for an image processing 
apparatus according to the present invention is charac- 
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terized by the following arrangement according to its 
tenth aspect. 

[0101] That is, there is provided a control method for 
an image processing apparatus which comprises a 
detachable lens unit, image sensing means for sensing 
an object image formed by the lens unit, storage means 
for storing a plurality of images sensed by the image 
sensing means in association with each other, and 
instruction means for instructing attachment/detach- 
ment of the lens unit, comprising the step of: inhibiting 
the lens unit from being detached when the instruction 
means outputs an instruction for detaching the lens unit 
after a first one of the plurality of images to be stored in 
the storage means in association with each other is 
sensed and stored. 

[0102] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
eleventh aspect. 

[0103] That is, there is provided a control method for 
an image processing apparatus which comprises a 
detachable lens unit, image sensing means for sensing 
an object image formed by the lens unit, storage means 
for storing a plurality of images sensed by the image 
sensing means in association with each other, instruc- 
tion means for instructing attachment/detachment of the 
lens unit, and display means for displaying predeter- 
mined information on an external unit, comprising the 
step of: controlling the display means to display alert 
information and inhibiting the lens unit from being 
detached when the instruction means outputs an 
instruction for detaching the lens unit after a first one of 
the plurality of images to be stored in the storage means 
in association with each other is sensed and stored. 
[0104] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
twelfth aspect. 

[0105] That is, there is provided a control method for 
an image processing apparatus which comprises a 
detachable lens unit, image sensing means for sensing 
an object image formed by the lens unit, storage means 
for storing a plurality of images sensed by the image 
sensing means in association with each other, and 
instruction means for instructing attachment/detach- 
ment of the lens unit, comprising the step of: permitting 
detachment of the lens unit when the instruction means 
outputs an instruction for detaching the lens unit before 
a first one of the plurality of images to be stored in the 
storage means in association with each other is sensed 
and stored, and inhibiting the lens unit from being 
detached when the instruction means outputs an 
instruction for detaching the lens unit after a first one of 
the plurality of images to be stored in the storage means 
in association with each other is sensed and stored. 
[0106] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 



thirteenth aspect 

[0107] That is, there is provided a control method for 
an image processing apparatus which comprises a 
detachable lens unit having nonvolatile storage means, 
s image sensing means for sensing an object image 
formed by the lens unit, storage means for storing a plu- 
rality of images sensed by the image sensing means in 
association with each other, and instruction means for 
instructing attachment/detachment of the lens unit, 
io comprising the step of: permitting detachment of the 
lens unit after information pertaining to an operation 
state of the lens unit is stored in the nonvolatile storage 
means, when the instruction means outputs an instruc- 
tion for detaching the lens unit, and reading out informa- 
15 tion pertaining to a use state of the lens unit stored in 
the nonvolatile storage means and re-setting an opera- 
tion state of the image processing apparatus in accord- 
ance with the readout information, when the lens unit is 
attached again. 
20 [0108] A control method for an image processing 
apparatus according to the present invention is charac- 
terized by the following arrangement according to its 
fourteenth aspect. 

[0109] That is, there is provided a control method for 

25 an image processing apparatus which comprises a 
detachable lens unit having nonvolatile storage means, 
image sensing means for sensing an object image 
formed by the lens unit, storage means for storing a plu- 
rality of images sensed by the image sensing means in 

30 association with each other, and instruction means for 
instructing attachmentfdetachment of the lens unit, 
comprising the step of: permitting detachment of the 
lens unit after information pertaining to an operation 
state of the lens unit and user information are stored in 

35 the nonvolatile storage means, when the instruction 
means outputs an instruction for detaching the lens unit, 
and reading out information pertaining to a use state of 
the lens unit and the user information stored in the non- 
volatile storage means and re-setting an operation state 

40 of the image processing apparatus in accordance with 
the readout information in unit of users, when the lens 
unit is attached again. 

[0110] A storage medium according to the present 
invention is characterized by the following arrangement 

45 according to its eleventh aspect. 

[0111] That is. there is provided a storage medium 
that stores a control program for controlling an image 
processing apparatus which comprises a detachable 
lens unit, image sensing means for sensing an object 

so image formed by the lens unit, storage means tor stor- 
ing a plurality of images sensed by the image sensing 
means in association with each other, detection means 
for detecting attachment/detachment of the lens unit, 
and display means for displaying predetermined infor- 

55 mation on an external unit, the control program compris- 
ing a code of the step of: controlling the display means 
to display alert information when the detection means 
detects detachment of the lens unit after a first one of 
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the plurality of images to be stored in the storage means 
in association with each other is sensed and stored. 
[0112] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its twelfth aspect. s 
[0113] That is, there is provided a storage medium 
that stores a control program for controlling an image 
processing apparatus which comprises a detachable 
lens unit, image sensing means for sensing an object 
image formed by the lens unit, storage means for stor- w 
ing a plurality of images sensed by the image sensing 
means in association with each other, and detection 
means for detecting attachment/detachment of the lens 
unit, the control program comprising a code of the step 
of: starting sensing of a plurality of new images to be 15 
stored in association with each other when the detection 
means detects detachment of the lens unit after a first 
one of the plurality of images to be stored in the storage 
means in association with each other is sensed and 
stored. 2 o 
[0114] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its thirteenth aspect. 
[0115] That is, there is provided a storage medium 
that stores a control program for controlling an image 25 
processing apparatus which comprises a detachable 
lens unit, image sensing means for sensing an object 
image formed by the lens unit, storage means for stor- 
ing a plurality of images sensed by the image sensing 
means in association with each other, detection means 30 
for detecting attachment/detachment of the lens unit, 
and display means for displaying predetermined infor- 
mation on an external unit, the control program compris- 
ing a code of the step of: controlling the display means 
to display alert information and then starting sensing of 35 
a plurality of new images to be stored in association with 
each other when the detection means detects detach- 
ment of the lens unit after a first one of the plurality of 
images to be stored in the storage means in association 
with each other is sensed and stored. <o 
[0116] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its fourteenth aspect. 
[0117] That is, there is provided a storage medium 
that stores a control program for controlling an image 45 
processing apparatus which comprises a detachable 
lens unit, image sensing means for sensing an object 
image formed by the lens unit, storage means for stor- 
ing a plurality of images sensed by the image sensing 
means in association with each other, and instruction so 
means for instructing attachment/detachment of the 
lens unit, the control program comprising a code of the 
step of: inhibiting the lens unit from being detached 
when the instruction means outputs an instruction for 
detaching the lens unit after a first one of the plurality of 55 
images to be stored in the storage means in association 
with each other is sensed and stored. 
[0118] A storage medium according to the present 



invention is characterized by the following arrangement 
according to its fifteenth aspect. 
[0119] That is, there is provided a storage medium 
that stores a control program for controlling an image 
processing apparatus which comprises a detachable 
lens unit, image sensing means for sensing an object 
image formed by the lens unit, storage means for stor- 
ing a plurality of images sensed by the image sensing 
means in association with each other, instruction means 
for instructing attachment/detachment of the lens unit, 
and display means for displaying predetermined infor- 
mation on an external unit, the control program compris- 
ing a code of the step of: controlling the display means 
to display alert information and inhibiting the lens unit 
from being detached when the instruction means out- 
puts an instruction for detaching the lens unit after a first 
one of the plurality of images to be stored in the storage 
means in association with each other is sensed and 
stored. 

[0120] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its sixteenth aspect. 
[0121] That is. there is provided a storage medium 
that stores a control program for controlling an image 
processing apparatus which comprises a detachable 
lens unit, image sensing means for sensing an object 
image formed by the lens unit, storage means for stor- 
ing a plurality of images sensed by the image sensing 
means in association with each other, and instruction 
means for instructing attachment/detachment of the 
lens unit, the control program comprising a code of the 
step of: permitting detachment of the lens unit when the 
instruction means outputs an instruction for detaching 
the lens unit before a first one of the plurality of images 
to be stored in the storage means in association with 
each other is sensed and stored, and inhibiting the lens 
unit from being detached when the instruction means 
outputs an instruction for detaching the lens unit after a 
first one of the plurality of images to be stored in the 
storage means in association with each other is sensed 
and stored. 

[0122] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its seventeenth aspect. 
[0123] That is. there is provided a storage medium 
that stores a control program for controlling an image 
processing apparatus which comprises a detachable 
lens unit having nonvolatile storage means, image 
sensing means for sensing an object image formed by 
the lens unit, storage means for storing a plurality of 
images sensed by the image sensing means in associ- 
ation with each other, and instruction means for instruct- 
ing attachment/detachment of the lens unit, the control 
program comprising a code of the step of: permitting 
detachment of the lens unit after information pertaining 
to an operation state of the lens unit is stored in the non- 
volatile storage means, when the instruction means out- 
puts an instruction for detaching the lens unit, and 
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reading out information pertaining to a use state of the 
lens unit stored in the nonvolatile storage means and re- 
setting an operation state of the image processing 
apparatus in accordance with the readout information, 
when the lens unit is attached again. s 
[0124] A storage medium according to the present 
invention is characterized by the following arrangement 
according to its eighteenth aspect. 
[0125] That is. there is provided a storage medium 
that stores a control program for controlling an image 10 
processing apparatus which comprises a detachable 
lens unit having nonvolatile storage means, image 
sensing means for sensing an object image formed by 
the lens unit, storage means for storing a plurality of 
images sensed by the image sensing means in associ- is 
ation with each other, and instruction means for instruct- 
ing attachment/detachment of the lens unit, the control 
sxogram comprising a code of the step of: permitting 
detachment of the lens unit after information pertaining 
to an operation state of the lens unit and user informa- 20 
t*on are stored in the nonvolatile storage means, when 
tr>e instruction means outputs an instruction for detach- 
ing the lens unit, and reading out information pertaining 
to a use state of the lens unit and the user information 
stored in the nonvolatile storage means and re-setting 25 
an operation state of the image processing apparatus in 
accordance with the readout information in unit of users, 
when the lens unit is attached again. 
[0126] Other objects and advantages besides those 
discussed above shall be apparent to those skilled in 30 
the art from the description of a preferred embodiment 
of the invention which follows. In the description, refer- 
ence is made to accompanying drawings, which form a 
part hereof, and which illustrate an example of the 
invention. Such example, however, is not exhaustive of 35 
the various embodiments of the invention, and therefore 
reference is made to the claims which follow the 
description for determining the scope of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 40 

[0127] 

Fig. 1 is a block diagram showing the arrangement 
of an image processing apparatus according to the 45 
first embodiment of the present invention; 
Fig. 2 is a flow chart of a main routine of the image 
processing apparatus according to the first embod- 
iment; 

Fig. 3 is a flow chart of the main routine of the so 
image processing apparatus according to the first 
embodiment; 

Fig. 4 is a flow chart of the main routine of the 
image processing apparatus according to the first 
embodiment; 55 
Fig. 5 is a flow chart of a distance measure- 
ment/photometry process routine of the image 
processing apparatus; 



Fig. 6 is a flow chart of an image sensing process 
routine of the image processing apparatus; 
Fig. 7 is a flow chart of a display image process rou- 
tine of the image processing apparatus; 
Fig. 8 is a flow chart of a compression process rou- 
tine of the image processing apparatus; 
Fig. 9 is an explanatory view of a panoramic image 
sensing mode setup window of the image process- 
ing apparatus; 

Figs. 10A and 10B are explanatory views of an ini- 
tial horizontal panoramic mode window of the 
image processing apparatus; 
Figs. 1 1 A and 1 1B are explanatory views of an ini- 
tial vertical panoramic mode window of the image 
processing apparatus; 

Figs. 12A and 12B are explanatory views of a 2 x 2 
panoramic mode image sensing window of the 
image processing apparatus; 
Figs. 13A and 13B are explanatory views of a 2 x 2 
panoramic mode image sensing window of the 
image processing apparatus; 
Figs. 14A and 14B are explanatory views of a hori- 
zontal panoramic mode image sensing window of 
the image processing apparatus; 
Figs. 15A and 15B are explanatory views of a verti- 
cal panoramic mode image sensing window of the 
image processing apparatus; 
Fig. 1 6 is a block diagram showing the arrangement 
of an image processing apparatus according to the 
second embodiment of the present invention; 
Fig. 17 is a flow chart of a main routine showing the 
operation of the image processing apparatus 
according to the second embodiment; 
Fig. 18 is a flow chart of the main routine showing 
the operation of the image processing apparatus 
according to the second embodiment; 
Fig. 19 is a flow chart of the main routine showing 
the operation of the image processing apparatus 
according to the second embodiment; 
Fig. 20 is a flow chart of the main routine showing 
the operation of the image processing apparatus 
according to the second embodiment; 
Fig. 21 is a flow chart of the main routine showing 
the operation of the image processing apparatus 
according to the second embodiment; 
Fig. 22 is a flow chart of a main routine according to 
the third embodiment of the present invention; 
Fig. 23 is a flow chart of the main routine according 
to the third embodiment of the present invention; 
Fig. 24 is a flow chart of the main routine according 
to the third embodiment of the present invention; 
Fig. 25 is a flow chart of the main routine according 
to the third embodiment of the present invention; 
Fig. 26 is a flow chart of the main routine according 
to the third embodiment of the present invention; 
Fig. 27 is a flow chart of a main routine showing the 
operation of an image processing apparatus 
according to the fourth embodiment of the present 
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invention; 

Fig. 28 is a flow chart of the main routine showing 
the operation of an image processing apparatus 
according to the fourth embodiment of the present 
invention; 

Fig. 29 is a flow chart of the main routine showing 
the operation of an image processing apparatus 
according to the fourth embodiment of the present 
invention; 

Fig. 30 is a flow chart of the main routine showing 
the operation of an image processing apparatus 
according to the fourth embodiment of the present 
invention; 

Fig. 31 is a block diagram showing the arrangement 

of an image processing apparatus according to the 

fifth embodiment of the present invention; 

Fig. 32 is a flow chart of a main routine of the fifth 

embodiment; 

Fig. 33 is a flow chart of the main routine of the fifth 
embodiment; 

Fig. 34 is a flow chart of the main routine of the fifth 
embodiment; 

Fig. 35 is a flow chart of the main routine of the fifth 
embodiment; and 

Fig. 36 is a flow chart of the main routine of the fifth 
embodiment. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0128] The preferred embodiments of the present 
invention will be described in detail hereinafter with ref- 
erence to the accompanying drawings. 

(First Embodiment) 

[0129] Fig. 1 is a block diagram showing the arrange- 
ment of an image processing apparatus according to 
the first embodiment. Referring to Fig. 1, reference 
numeral 100 denotes an image processing apparatus; 
and 101, a photographing lens used for sensing an 
object image. Reference numeral 102 denotes a shutter 
with a stop function. Reference numeral 103 denotes an 
image sensing element for converting an optical image 
into an electrical signal. Reference numeral 104 
denotes an A/D converter for converting an analog sig- 
nal output from the image sensing element 103 into a 
digital signal. Reference numeral 105 denotes a timing 
generation circuit for supplying clock signals and control 
signals to the image sensing element 103, the A/D con- 
verter 104, and a D/A converter 109 (to be described 
later). The timing generation circuit 105 is controlled by 
a memory control circuit 107 and system control circuit 
1 1 8 (to be described later) . 

[0130] Reference numeral 106 denotes an image 
processing circuit which performs predetermined pixel 
interpolation and color conversion for data supplied 
from the A/D converter 104 or the memory control cir- 



cuit 107 (to be described later). The image processing 
circuit 106 makes a predetermined computation using 
sensed image data to execute a TTL (Through The 
Lens) AF (auto focus) process. AE (auto exposure) 

5 process, and EF (flash pre-emission) process in which 
the system control circuit 1 18 controls an exposure con- 
trol means 113 and distance measurement control 
means 114 (to be described later) on the basis of the 
obtained computation result. Furthermore, the image 

w processing circuit 106 makes a predetermined compu- 
tation using sensed image data, and executes a TTL 
AWB (auto white balance) process on the basis of the 
obtained computation result. 

[0131] Reference numeral 107 denotes the memory 
75 control circuit, which controls the A/D converter 1 04, the 
timing generation circuit 105, the image processing cir- 
cuit 106, an image display unit 110 (to be described 
later), the D/A converter 109 (to be described later), a 
memory 111 (to be described later), and a compres- 
20 sion/expansion circuit 32 (to be described later). Data 
output from the A/D converter 104 is written in an image 
display memory 1 08 or memory 111 (to be described 
later) via the image processing circuit 106 and memory 
control circuit 107 or directly via the memory control cir- 
25 cuit 107. 

[01 32] Reference numeral 1 08 denotes the image dis- 
play memory. Reference numeral 109 denotes the D/A 
converter for converting a digital signal into an analog 
signal. Reference numeral 110 denotes the image dis- 

30 play unit comprising a TFT- LCD (thin film transistor liq- 
uid crystal display) or the like. Display image data 
written in the image display memory 108 is displayed by 
the image display unit 1 10 via the D/A converter 109. 
When sensed image data are displayed at image sens- 

35 ing timings using the image display unit 110. an elec- 
tronic viewfinder function can be implemented. The 
image display unit 1 1 0 can arbitrarily turn on/off its dis- 
play in response to an instruction from the system con- 
trol circuit 1 18 (to be described later). When the display 

40 is turned off, consumption power of the image process- 
ing apparatus 100 can be greatly reduced. 
[0133] Reference numeral 1 1 1 denotes the memory 
for storing a still image and moving image. The memory 

1 1 1 has a storage capacity large enough to store a pre- 
45 determined number of still images or a moving image 

for a predetermined period of time. With this arrange- 
ment, even in a continuous or panoramic image sensing 
mode for continuously sensing a plurality of still images, 
many images can be written in the memory 1 1 1 at high 
so speed. Also, the memory 1 1 1 can be used as a work 
area of the system control circuit 118 (to be described 
later). 

[01 34] Reference numeral 1 12 denotes the compres- 
sion/expansion circuit for compressing/expanding 
55 image data by adaptive discrete cosine transformation 
(ADCT) or the like. The compression/expansion circuit 

112 reads an image stored in the memory 111, com- 
presses or expands it, and writes the processed data in 
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the memory 111. Reference numeral 113 denotes the 
exposure control means for controlling the shutter 102. 
The exposure control means 113 also has a flash light 
control function in cooperation with an electronic flash 
117 (to be described later). Reference numeral 114 
denotes the distance measurement control means for 
controlling focusing of the photographing lens 101 . Ref- 
erence numeral 1 15 denotes a zoom control means for 
controlling zooming. Reference numeral 116 denotes a 
barrier control means for controlling operation of a pro- 
tection means (to be described later) serving as a bar- 
rier. Reference numeral 117 denotes the electronic 
flash which has a function of projecting AF auxiliary light 
and the flash light control function. 
[01 35] The exposure control means 113 and distance 
measurement control means 114 are controlled using 
TTL, and the system control circuit 118 controls the 
exposure control means 113 and distance measure- 
ment control means 1 14 on the basis of the computation 
result of the image processing circuit 106 using sensed 
image data. 

[01 36] Reference numeral 1 1 8 denotes the system 
control circuit for controlling the overall image process- 
ing apparatus 100. Reference numeral 119 denotes a 
memory for storing constants, variables, programs, and 
the like for operations of the system control circuit 118. 
Reference numeral 120 denotes an indication unit 
which includes a liquid crystal display device, loud- 
speaker, and the like for indicating operation states, 
messages, and the like using characters, images, 
voices, and the like in accordance with execution of a 
program by the system control circuit 118. The indica- 
tion unit 120 is placed at one or a plurality of easy-to- 
see positions around the console of the image process- 
ing apparatus 1 00, and is comprised of a combination of 
an LCD (liquid crystal display device), LEDs (light-emit- 
ting diodes), tone generation element, and the like. 
Some functions of the indication unit 120 are placed 
within an optical viewfinder 138. 
[0137] Of the indication contents of the indication unit 
120, those indicated on an LCD or the like include, e.g., 
a single shot/continuous shot indication, self -timer indi- 
cation, compression ratio indication, recording pixel 
count indication, recording image count indication, 
remaining photographable image count indication, shut- 
ter speed indication, aperture value indication, expo- 
sure correction indication, flash indication, red-eye 
suppression indication, macro-image sensing indica- 
tion, buzzer setting indication, timepiece battery remain- 
ing capacity indication, error indication, information 
indication using numerals having a plurality of digits, 
attached/detached state indication of recording media 
200a and 200b. communication l/F operation indication, 
date/time indication, indication that indicates a connec- 
tion state with an external computer, and the like. 
[0138] Of the indication contents of the indication unit 
120. those indicated within the optical viewfinder 138 
include, e.g., an in-focus indication, image sensing 



ready indication, camera shake alert indication, flash 
charging indication, flash charging completion indica- 
tion, shutter speed indication, aperture value indication, 
exposure correction indication, recording medium write 

5 access indication, and the like. 

[01 39] Furthermore, of the indication contents of the 
indication unit 120. those indicated by LEDs or the like 
include, e.g.. in-focus indication, image sensing ready 
indication, camera shake alert indication, flash charging 

io indication, flash charging completion indication, record- 
ing medium write access indication, macro-image sens- 
ing setting notification indication, secondary battery 
charged state indication, and the like. 
[0140] Of the indication contents of the indication unit 

is 1 20, those to be indicated by lamps include, e.g., a self- 
timer notification lamp, and the like. The self-timer noti- 
. f ication lamp may be commonly used as that for emitting 
AF auxiliary light. 

[0141] Reference numeral 121 denotes a rewritable 
20 nonvolatile memory, which uses, e.g., an EEPROM 
(electrically erasable and programmable read only 
memory). 

[0142] Reference numerals 122, 123, 124, 125. 126. 
127. and 128 denote operation means for inputting var- 

25 ious operation instructions of the system control circuit 
1 18. These operation means are constructed by one or 
a plurality of combinations of a switch, a dial, a touch 
panel, a pointer using line of sight detection, a voice rec- 
ognition device, and the like. 

30 [01 43] Examples of these operation means 1 22 to 1 28 
will be explained below. 

[0144] Reference numeral 122 denotes a mode dial 
switch, which can select various function mode posi- 
tions such as power OFF, automatic image sensing 
35 mode, image sensing mode, panoramic image sensing 
mode, playback mode, multi -frame playback/erase 
mode, PC (personal computer connection mode, and 
the like. 

[01 45] Reference numeral 1 23 denotes a first shutter 

40 switch, which is turned at the half stroke position of a 
shutter button (not shown), and instructs start of an AF 
(auto focus) process. AE (auto exposure) process, AWB 
(auto white balance) process, EF (flash pre-emission) 
process, and the like. 

45 [0146] Reference numeral 124 denotes a second 
shutter switch, which is turned on upon completion of 
operation of the shutter button (not shown), and 
instructs start of a series of processes including an 
exposure process for writing a signal read out from the 

so image sensing element 103 as image data in the mem- 
ory 111 via the A/D converter 104 and memory control 
circuit 107, a development process using computation 
results in the image processing circuit 106 and memory 
control circuit 107. and a recording process for reading 

55 out image data from the memory 111. compressing the 
readout data by the compression/expansion circuit 1 1 2, 
and writing the compressed image data in the recording 
medium 200a or 200b. 
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[0147] Reference numeral 125 denotes selec- 
tion/change switch that can select and change various 
functions upon executing image sensing and playback 
in, e.g., the panoramic mode. 

[0148] Reference numeral 126 denotes a determina- 
tion/execution switch which can determine and execute 
various functions upon executing image sensing and 
playback in, e.g., the panoramic mode. 
[0149] Reference numeral 127 denotes a console 
including various buttons, touch panel, and the like, 
which include a menu button, set button, macro button, 
multi-frame playback new page button, flash setting but- 
ton, single shot/continuous shot/self-timer switch but- 
ton, menu movement + (plus) button, menu movement - 
(minus) button, playback image movement + (plus) but- 
ton, playback image movement - (minus) button, sensed 
image quality selection button, exposure correction but- 
ton, date/time setting button, image display ON/OFF 
button for turning on/off the image display unit 110, 
quick review ON/OFF setting button for setting a quick 
review function of automatically playing back sensed 
image data on the image display unit 110 immediately 
after image sensing, and the like. 
[0150] Reference numeral 128 denotes a compres- 
sion mode switch which is used for selecting the com- 
pression ratio of JPEG compression or selecting a 
CCDRAW mode for directly recording a signal output 
from the image sensing element 103 and converted into 
a digital signal on the recording medium 200a or 200b 
(to be described later. 

[01 51 ] The JPEG compression mode includes, e.g. , a 
normal mode and fine mode. In the JPEG compression 
mode, image data, which is read out from the image 
sensing element 103 and is written in the memory 1 1 1 
via the A/D converter 104, image processing circuit 106. 
and memory control circuit 107, is read out, and is com- 
pressed at the selected compression ratio by the com- 
pression/expansion circuit 112. After that, the 
compressed image data is recorded on the recording 
medium 200a or 200b. 

[0152] In the CCDRAW mode, image data, which is 
directly read out in units of lines in correspondence with 
the pixel formats of color filters of the image sensing 
element 103, and is written in the memory 111 via the 
A/D converter 104 and memory control circuit 107, is 
r ad out. and is recorded on the recording medium 
200a or 200b. 

[0153] Reference numeral 129 denotes a power sup- 
ply control means, which is comprised of a battery 
detection circuit, a DC-DC converter, a switch circuit for 
switching a block to be energized, and the like. The 
power supply control means 129 detects the pres- 
ence/absence, type, and remaining battery amount of a 
battery attached, controls the DC-DC converter on the 
basis of such detection results and an instruction from 
the system control circuit 1 18, and supplies a required 
voltage to the respective units including the recording 
media 200a and 200b for a required period of time. 



[0154] Reference numerals 130 and 131 denote con- 
nectors for connecting the power supply means 129 and 
a power supply 132 (to be described below). Reference 
numeral 132 denotes the power supply, which includes 

s a primary battery such as an alkali battery, lithium bat- 
tery, or the like, a secondary battery such as an NiCd 
battery. NiMH battery, U Battery, or the like, an AC 
adapter, and the like. Reference numerals 133 and 134 
denote interfaces (l/Fs) with recording media 200a and 

jo 200b (to be described later); and 1 35 and 136, connec- 
tors for connecting the interfaces (l/Fs) with the record- 
ing media 200a and 200b (to be described later). 
[0155] Reference numeral 137 denotes a recording 
medium attachment/detachment detection means for 

15 detecting whether or not the recording medium 200a or 
200b (to be described later) is attached to the connector 
135 or 136. The recording medium attachment/detach- 
ment detection means 137 can also detect whether or 
not one of devices other than the recording medium 

20 200a or 200b (to be described later) such as various 
communication cards, and the like (to be described 
later) is attached. Note that this embodiment has two 
sets of interfaces and connectors that receive the 
recording media 200a and 200b. Of course, the number 

25 of sets of interfaces and connectors that receive the 
recording media is not particularly limited. 
[01 56] Also, combinations of interfaces and connec- 
tors of different standards may be used. As the interface 
and connector, those complying with the standards of a 

30 PCMCIA card, CF (compact flash) card, and the like 
may be used. 

[01 57] Furthermore, when the interfaces 1 33 and 1 34, 
and connectors 135 and 136 use those complying with 
the standards of a PCMCIA card, CF (compact flash) 

35 card, and the like, various communication cards such as 
a LAN card, modem card, USB card, IEEE1394 card, 
P1284 card. SCSI card, PHS. and the like are con- 
nected thereto, image data and associated manage- 
ment information can be transferred between the image 

40 processing apparatus 1 00 and an external computer or 
its peripheral devices such as a printer and the like. 
[01 58] Reference numeral 1 50 denotes the protection 
means serving as a barrier which covers an image 
sensing unit S including the photographing lens 101 of 

45 the image processing apparatus 1 00 to protect it from 
contamination and damages. Reference numeral 138 
denotes the optical viewfinder, which allows image 
sensing using the optical viewfinder alone without using 
the electron^ viewfinder function implemented by the 

so image display unit 110. in the optica! viewfinder 138, 
some functions of the indication unit 120, e.g.. an in- 
focus indication, camera shake alert indication, flash 
charging indication, shutter speed indication, aperture 
value indication, exposure correction indication, and the 

55 like are placed. 

[0159] Reference numeral 139 denotes a communica- 
tion means having various communication functions 
such as RS232C. USB. IEEE1394. P1284, SCSI. 
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modem, LAN, radio communication, and the like. Refer- 
ence numeral 140 denotes a connector or antenna, 
which serves as a connector when the image process- 
ing apparatus 100 is connected to another device using 
the communication means 139, or serves as an 5 
antenna in case of radio communications. 
[01 60J Reference numerals 200a and 200b denote the 
recording media such as a memory card, hard disk, or 
the like. The recording media 200a and 200b respec- 
tively comprise recording units 201a and 201b each 10 
comprised of a semiconductor memory, magnetic disk, 
or the like, interfaces (l/Fs) 202a and 202b with the 
image processing apparatus 100. and connectors 203a 
and 203b for connecting the image processing appara- 
tus 100. 15 
[01 61 ] The operation of the image processing appara- 
tus 100 according to this embodiment with the afore- 
mentioned arrangement will be explained below using 
Figs. 2 to 8. 

[0162] Figs. 2 to 4 are flow charts showing the main 20 
routine of the image processing apparatus 100 accord- 
ing to this embodiment. 

[01 63] The operation will be explained first using Figs. 
2 to 4. 

[01 64] Upon power ON after battery exchange or the 25 
like, the system control circuit 118 initializes flags, con- 
trol variables, and the like and also initializes (initially 
sets up) the respective units of the image processing 
apparatus 100 in step S201 in Fig. 2. In step S202, the 
system control circuit 1 18 checks the setting position of 30 
the mode dial switch 122. If the mode cfial switch 122 is 
set at a power OFF position, the system control circuit 
118 executes a predetermined process in step S213. 
More specifically, the system control circuit 118 
changes the indications of the respective indication 35 
units to an end state, protects the image sensing unit S 
by closing the barrier of the protection means 150, 
records required parameters and setting values includ- 
ing flags, control variables, and the like, and the setting 
mode in the nonvolatile memory 121, cuts off unneces- 40 
sary power supply to the respective units of the image 
processing apparatus 100 including the image display 
unit 1 10 by the power supply control means 129, and so 
forth. After that, the flow returns to step S202. 
[0165] If it is determined in step S202 that the mode 45 
dial switch 122 is set at any of other mode positions, the 
system control circuit 1 18 checks using the power sup- 
ply control means 129 in step S203 if the remaining 
capacity and operation state of the power supply 132 
comprising batteries and the like pose any problem in so 
the operation of the image processing apparatus 100. If 
any problem is found, a predetermined alert indication is 
made by means of an image or voice using the indica- 
tion unit 120 in step S214, and the flow then returns to 
step S202. 55 
[0166] If no problem is found in step S203. the system 
control circuit 118 checks in step S204 if the operation 
state of the recording medium 200a or 200b poses any 



30 

problem in the operation of the image processing appa- 
ratus 100, in particular, recording/playback of image 
data to/from the recording medium 200a or 200b. If any 
problem is found, a predetermined alert indication is 
made by means of an image or voice using the indica- 
tion unit 120 in step S214, and the flow then returns to 
step S202. 

[01 67] If no problem is found in step S204, various set- 
ting states of the image display apparatus 100 are indi- 
cated by means of an image or voice using the 
indication unit 120 in step S205. When the image dis- 
play of the image display unit 1 10 is ON. various setting 
states of the image display apparatus 100 are indicated 
by means of an image or voice also using the image dis- 
play unit 110. 

[01 68] The setting position of the mode dial switch 1 22 
is then checked in step S206. If the mode dial switch 
122 is set at an image sensing mode position, a prede- 
termined image sensing process is executed in step 
S215. and the flow then returns to step S202. H the 
mode dial switch 122 is set at any of other mode posi- 
tions, the system control circuit 118 executes a process 
corresponding to the selected (set) mode in step S216, 
and the flow returns to step S202. Furthermore, if the 
mode dial switch 122 is set at a panoramic mode posi- 
tion, the system control circuit 1 18 displays a panoramic 
mode selection window on the image display unit 1 10 in 
step S207. 

[0169] The user of the image processing apparatus 
100 can select an arbitrary one of a plurality of pano- 
ramic image sensing modes by observing the pano- 
ramic mode selection window displayed on the image 
display unit 110. 

[0170] The panoramic image sensing modes include 
a 2 x 2 panoramic mode, horizontal panoramic mode, 
vertical panoramic mode, and the like. In these modes, 
by combining a plurality of sensed images, a panoramic 
image, which is extended two-dimensionally, horizon- 
tally, or vertically, can be obtained. 
[01 71 ] A series of sensed images are appended with 
information indicating their positional relationship, and 
can be combined to generate a signal panoramic image 
by the image processing apparatus 100 itself or after 
they are transferred to, e.g., a computer or the like. 
[0172] Fig. 9 shows an example of the panoramic 
mode selection window. Referring to Fig. 9, reference 
numeral 901 denotes an icon for selecting the 2x2 pan- 
oramic mode used for sensing a two-dimensional pano- 
ramic image; 9Q2, an icon for selecting the horizontal 
panoramic mode used for sensing a horizontal pano- 
ramic image; and 903, an icon for selecting a vertical 
panoramic mode used for sensing a vertical panoramic 
image. 

[01 73] The user of the image processing apparatus 
100 can select a desired panoramic mode by the selec- 
tion/change switch 125 (e.g., [+] key), and can deter- 
mine that panoramic mode by the 
determination/execution switch 126 (e.g., [SET] key). 
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[01 74] Referring back to Fig. 2. upon completion of the 
process in step S207. the type of panoramic mode 
selected by the selection/change switch 125 and deter- 
mination/execution switch 126 is checked in step S208. 
If the 2 x 2 panoramic mode is selected, the system 5 
control circuit 1 18 initially sets up the 2 x 2 panoramic 
mode in step S209; if the vertical panoramic mode is 
selected, the system control circuit 118 initially sets up 
the vertical panoramic mode in step S210; or if the hor- 
izontal panoramic mode is selected, the system control 10 
circuit 118 initially sets up the horizontal panoramic 
mode in step S2 1 1 . More specifically, the system control 
circuit 1 1 8 sets initial values and memory areas of flags, 
parameters, and variables, reads out a display window 
serving as a user interface, and so forth in correspond- 15 
ence with the selected panoramic mode. After that, the 
system control circuit 118 displays a panoramic image 
sensing window corresponding to the selected mode on 
the image display unit 1 1 0 in step S21 2. and then enters 
the process shown in Fig. 3. 20 
[0175] Figs. 10A and 10B show examples of the hori- 
zontal panoramic mode image sensing window, Figs. 
1 1 A and 1 1 B show examples of the vertical panoramic 
mode image sensing window, and Figs. 12A and 12B 
show examples of the 2 x 2 panoramic mode image 25 
sensing window. 

[01 76] Fig. 1 0A shows an image sensing window that 
creates a panoramic window by continuously repeating 
image sensing to the right, and Fig. 10B shows an 
image sensing window that creates a panoramic win- 30 
dow by continuously repeating image sensing to the left. 
[0177] Referring to Figs. 10A and 10B. a through 
image from the electronic viewfinder is displayed on 
region "T to sense the first image of a series of pano- 
ramic images. Before completion of image sensing of 35 
the first image, rightward image sensing (Fig. 10A) or 
leftward image sensing (Fig. 10B) can be desirably 
selected using the selection/change switch 1 25 (e.g.. [+] 
key). The image sensing direction is determined by 
sensing the first image by pressing the shutter button in 40 
desired one of the display window states shown in Figs. 
1 0A and 1 0B. 

[0178] Note that the user can end image sensing in 
the horizontal panoramic mode by pressing the determi- 
nation/execution switch 126 (e.g., [SET] key). 45 
[01 79] Fig. 1 1 A shows an image sensing window that 
creates a panoramic window by continuously repeating 
image sensing upward, and Fig. 1 1 B shows an image 
sensing window that creates a panoramic window by 
continuously repeating image sensing downward. so 
[0180] Referring to Figs. 11A and 11B, a through 
image from the electronic viewfinder is displayed on 
region "1" to sense the first image of a series of pano- 
ramic images. Before completion of image sensing of 
the first image, upward image sensing (Fig. 11 A) or 55 
downward image sensing (Fig. 11B) can be desirably 
selected using the selection/change switch 1 25 (e.g.. [+] 
key). The image sensing direction is determined by 



sensing the first image by pressing the shutter button in 
desired one of the display window states shown in Figs. 
IIAand 11B. 

[01 81 ] Note that the user can end image sensing in 
the vertical panoramic mode by pressing the determina- 
tion/execution switch 126 (e.g., [SET] key). 
[0182] Fig. 12A shows an image sensing window for 
the first image upon creating a 2 x 2 panoramic window 
by two-dimensionalhy repeating image sensing of a total 
of four images in the vertical and horizontal directions, 
and Fig. 12B shows an image sensing window for the 
second image. 

[0183] Referring to Figs. 12A and 12B, a through 
image from the electronic viewfinder is displayed on, 
e.g., an upper right region to sense the first one of pan- 
oramic images. 

[0184] Upon completion of image sensing of the first 
image, the playback image of the sensed image is dis- 
played on, e.g.. the upper right region, and a through 
image from the electronic viewfinder is displayed on, 
e.g., the lower right region to sense the second image. 
The image sensing region of the electronic viewfinder is 
switched in turn to the next region after each image 
sensing. However, image sensing may be started from 
an arbitrary region, or may be redone by returning to the 
already sensed region. For this purpose, the image 
sensing region can be desirably shifted using the selec- 
tion/change switch 125 (e.g.. [+] switch). The image 
sensing region to be re-sensed is determined and 
image sensing is executed by pressing the shutter but- 
ton in a display window state that selects a desired 
region. Note that the user can end image sensing in the 
2x2 panoramic mode by pressing the determina- 
tion/execution switch 126 (e.g., [SET] key). 
[0185] Referring to Fig. 3, the system control circuit 
118 initializes flags and variables set in its internal 
memory or the memory 1 1 9 in step S301 . For example, 
the system control circuit 1 18 resets a panoramic image 
sensing start flag, sets a variable n and initializes it to n 
= 1 in step S302, and sets a variable m and initializes it 
to m = 1 in step S303. 

[0186] Note that the panoramic image sensing start 
flag is reset to indicate that no panoramic image sens- 
ing processes have been done. Also, the variable n indi- 
cates the position of the current image sensing region in 
image sensing of each panoramic mode. The variable m 
is used for returning the image sensing region to the 
immediately preceding region or canceling such return, 
upon executing image sensing again in the horizontal 
and vertical panoramic modes. An example of the 
method of using these flag and variables will be 
explained later. 

[0187] In step S304. the setting position of the mode 
dial switch 1 22 is checked. If the mode dial switch 1 22 is 
set at any of mode positions other than the panoramic 
mode, the system control circuit 1 18 ends image sens- 
ing in the panoramic mode, and the flow returns to step 
S202 in Fig. 2. In this manner, the user of the image 
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processing apparatus 100 can end the panoramic mode 
anytime he or she wants, and can select another mode. 
[0188] If the mode dial switch 122 is set at the pano- 
ramic mode position, it is checked in step S305 if end of 
the panoramic mode is selected. If end of the panoramic s 
mode is selected by pressing the determination/execu- 
tion switch 126 (e.g., [SET] key) while the mode dial 
switch 122 is kept set at the panoramic mode position, 
i.e., the display window keeps displaying the corre- 
sponding panoramic image sensing window, a pano- io 
ramie mode end confirmation message is displayed, 
and it is checked in step S314 if confirmation of the end 
is selected. If confirmation of end is selected, the flow 
returns to step S207 in Fig. 2; otherwise, the flow 
returns to step S304. rs 
[01891 If it is determined in step S305 that end of the 
panoramic mode is not selected, it is checked in step 
S306 if the selection/change switch 125 (e.g., [+] key) 
has been pressed while the image display unit 110 
keeps displaying the panoramic image sensing window. 20 
If the selection/change switch 125 (e.g.. [+] key) has 
been pressed, the system control circuit 118 discrimi- 
nates the contents of the panoramic mode in step S307. 
[0190] If the contents of the panoramic mode indicate 
the 2 x 2 panoramic mode, the system control circuit 25 
1 1 8 sets the variable n to be n = n + 1 in step S308. and 
checks in step S309 if n = 5. If n = 5. the system control 
circuit 118 sets n = n - 4 in step S310 to select the first 
image sensing region. After that, the flow advances to 
stepS311. 30 
[0191] More specifically, every time the selec- 
tion/change switch 125 (e.g., [+] key) is pressed, the 
image sensing region of interest shifts to one of the four 
image sensing regions in the 2 x 2 panoramic mode. For 
example, the image sensing region of interest shifts in 35 
the order of the upper right region, lower right region, 
lower left region, upper left region, upper right region, 
lower right region 

[0192] On the other hand, if it is determined in step 

5306 that the selection/change switch 1 25 (e.g. , [+] key) 40 
is not pressed, the control skips steps S307 to S310, 
and advances to step S31 1 . 

[0193] On the other hand, if it is determined in step 

5307 that the contents of the panoramic mode indicate 
the horizontal or vertical panoramic mode, the status of 45 
the image sensing start flag is checked in step S3 15. If 
the image sensing start flag is reset, i.e., no images 
have been sensed in a series of panoramic image sens- 
ing processes, the system control circuit 1 1 8 reverses 

the image sensing direction of the selected panoramic so 
mode in step S3 16, and the flow then advances to step 
S311. 

[01 94] If it is determined in step S31 5 that the image 
sensing start flag is reset, i.e.. the first image has 
already been sensed in a series of panoramic image ss 
sensing processes, the system control circuit 118 com- 
putes m = m x (-1), the sign of which changes like -1, 
+1, -1. +1.... in every computation in step S317, and 



also computes n = n + m for the variable n, which 
changes like n = n-1.n = n+1,n=n-l,n=n + 1,... 
in every computation in step S3 18. After that, the flow 
advances to step S31 1 . 

[0195] If n * 5 in step S309. the control skips step 
S310 and advances to step S311. 
[0196] With this control, movement to the already 
sensed region (which is limited to the lastly sensed one) 
or cancel that movement can be repetitively selected to 
sense an image again. 

[0197] After panoramic image sensing of the n-th 
image region, the user presses the selection/change 
switch 125 (e.g., [+] key) to confirm if quick review 
images of the (n-1)-th and n-th image regions smoothly 
join at their boundary regions. Then, the user temporar- 
ily releases (cancels) the selection/change switch 125 
(e.g., [+] key) to display a through image on the n-th 
image region, and presses the selection/change switch 
125 (e.g., [+] key) once again to display the original 
quick review image on the n-th image region and a 
through image on the (n+1)-th image region, thus exe- 
cuting the next panoramic image sensing. 
[0198] After panoramic image sensing of the n-th 
image region, upon pressing the selection/change 
switch 125 (e.g., [+] key) to confirm if quick review 
images of the (n-1)-th and n-th image regions smoothly 
join at their boundary regions, if the user determines 
that image sensing must be redone, he or she tempo- 
rarily releases the selection/change switch 125 (e.g., [+] 
key) to display a through image on the n-th image 
region, and presses the shutter switch 123 in that win- 
dow state to replace the image with the newly sensed 
image. Then, the user can execute the next panoramic 
image sensing. 

[01 99] The system control circuit 1 1 8 selects the n-th 
image region in step S31 1, and checks in step S312 if 
the window layout must be changed (switched) in corre- 
spondence with the selected image region. If the win- 
dow layout must be changed (switched) in 
correspondence with the selected image region, the 
system control circuit 118 switches and displays a new 
window layout in step S313, and then enters the proc- 
ess shown in Fig. 4. 

[0200] On the other hand, if the window layout need 
not be changed (switched) in correspondence with the 
selected image region, the control skips step S3 13 and 
enters the process shown in Fig. 4. With this control, a 
window corresponding to the reversed image sensing 
direction, a window corresponding to the changed 
image sensing region, a window that has updated char- 
acters and images for comments, or the like can be dis-_ 
played as needed. 

[0201] Upon sensing the second and subsequent 
images, the image sensing regions that display already 
sensed images are set in a quick review display state, 
and the image sensing region selected for the next 
image sensing is set in a through display state, thus dis- 
playing a playback image of the already sensed image 



18 

4SDOCID: <EP 0942598A2J_> 



35 



EP 0 942 598 A2 



36 



and the next image to be sensed to partially overlap 
each other. In this manner, the user of the image 
processing apparatus 100 can easily determine the next 
image sensing angle. 

[0202] Referring to Fig. 4, the system control circuit 5 
1 18 sets a through display state that displays sensed 
image data in turn in step S401 . 
[0203] In through display, data, which have been writ- 
ten in the image display memory 108 via the image 
sensing element 103, A/D converter 104, image w 
processing circuit 106, and memory control circuit 107. 
are displayed on the aforementioned image sensing 
region of the image display unit via the memory control 
circuit 107 and D/A converter 109. thereby implement- 
ing an electronic viewfinder function. is 
[0204] The system control circuit 108 checks in step 
S402 if the first shutter switch 123 has been pressed. If 
the first shutter switch 123 has not been pressed, the 
flow returns to step S304 in Fig. 3. On the other hand, if 
the first shutter switch 123 has been pressed, the sys- 20 
tern control circuit 118 sets the display state of the 
image display unit 110 in a freeze display state in step 
S403. 

[0205] In freeze display, image data on the image dis- 
play memory 108 is inhibited from being rewritten via 25 
the image sensing element 103. A/D converter 104, 
image processing circuit 106. and memory control cir- 
cuit 107, and the latest written image data is displayed 
on the above-mentioned image display region of the 
image display unit 110 via the memory control circuit 30 
107 and D/A converter 109, thus displaying a frozen 
image on the electronic viewfinder. 
[0206] In step S404, the system control circuit 1 18 
executes distance measurement to adjust the focus of 
the photographing lens 101 on the object, and executes 35 
photometry to determine the aperture value and shutter 
speed. In photometry, the ON/OFF state of the elec- 
tronic flash 1 1 7 is set if necessary. The distance meas- 
urement/photometry process in step S404 will be 
explained later with reference to Fig. 5. 40 
[0207] Upon completion of the distance measure- 
ment/photometry process in step S404, the system con- 
trol circuit 1 1 8 sets the display state of the image display 
unit 110 in a through display state in step S405. Note 
that this through display state is the same operation 45 
state as that in step S401. 

[0208] The system control circuit 118 checks in step 
S406 if the second shutter switch 124 has been 
pressed. If the second shutter switch 124 has not been 
pressed, the control circuit 118 checks in step S417 rf so 
the first shutter switch 1 23 is canceled, rf the first shutter 
switch 123 is canceled, the fbw returns to step S406; 
otherwise, the flow returns to step S304 in Fig. 3. 
[0209] If it is determined in step S406 that the second 
shutter switch 124 has been pressed, the system con- 55 
trol circuit 1 1 8 sets the display state of the image display 
unit 110 in a fixed-color display state in step S407. In 
fixed-color display, in place of sensed image data, which 



has been written in the image display memory 108 via 
the image sensing element 103. A/D converter 104, 
image processing circuit 106, and memory control cir- 
cuit 107, substitute fixed-color image data is displayed 
on the image display unit 110 via the memory control 
circuit 107 and D/A converter 109, thus displaying a 
fixed-color image on the electronic viewfinder. 
[0210] In step S408, the system control circuit 118 
executes an image sensing process including an expo- 
sure process for writing sensed image data in the mem- 
ory 111 via the image sensing element 103, A/D 
converter 104, image processing circuit 106, and mem- 
ory control circuit 107 or directly from the A/D converter 
104 via the memory control circuit 107, and a process 
for executing various processes by reading out image 
data written in the memory 1 1 1 using the memory con- 
trol circuit 107 and image processing circuit 106 if 
required. The image sensing process will be described 
in detail later with reference to Fig. 6. 
[0211] In step S409, the system control circuit 118 
executes a display image process for reading out image 
data written in the memory 1 1 1 by the image sensing 
process in step S408. and transferring the readout data 
to the image display memory 108 via the memory con- 
trol circuit 107. The display image process will be 
explained in detail later with reference to Fig. 7. 
[0212] In step S410. the system control circuit 118 
sets the display state of the image display unit 1 10 in a 
quick review display state. In quick review display, 
image data, which has been transferred to the image 
display memory 108 by the display image process in 
step S409, is displayed on the above-mentioned image 
sensing region of the image display unit 110 via the 
memory control circuit 107 and D/A converter 109, thus 
implementing an electronic viewfinder function that 
automatically plays back the sensed image. 
[0213] In step S411, the system control circuit 118 
executes a compression process in which various 
image processes are done using the memory control 
circuit 107 and image processing circuit 106 if required 
by reading out sensed image data written in the mem- 
ory 1 1 1, or an image is compressed in correspondence 
with the selected mode using the compression/expan- 
sion circuit 112. The compression process will be 
explained in detail later with reference to Fig. 8. 
[0214] In step S412, the system control circuit 118 
executes a recording process for writing image data in 
the recording medium 200a or 200b such as a memory 
card, compact flash card, or the like via the interface 
133 or 134 and the connector 135 or 136. 
[021 5] When the image display unit 1 1 0 is ON. a mes- 
sage such as "BUSY" or the like that indicates that a 
write is in progress is displayed on the image display 
unit 110 during a write of image data on the recording 
medium 200a or 200b. Furthermore, the indication unit 
120 makes a recording medium write access indication 
by flickering an LED or the like in combination with the 
above message. 
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[021 6] The system control circuit 1 1 8 continues quick 
review display on the image display unit 110 until the 
second shutter switch 124 is canceled. With this display, 
the user can carefully check the sensed image while 
pressing the second shutter switch 124. 
[0217] The system control circuit 1 18 checks in step 
S413 if the second shutter switch 124 is canceled, until 
it is actually canceled. If the second shutter switch 124 
is canceled, the system control circuit 1 18 sets the pan- 
oramic image sensing start flag in step S414 to indicate 
that the first image in a series of panoramic image sens- 
ing processes has already been sensed. In this manner, 
as has been described in step S315 in Fig. 3, the image 
sensing direction can be inhibited from being reversed 
after the first image is sensed in the horizontal or verti- 
cal panoramic mode. 

[021 8] The system control circuit 1 1 8 sets the variable 
n to be n = n + 1 to select the next image sensing region 
in step S415. and resets the variable m to m = 1 in step 
S416. Then, the flow returns to step S304 in Fig. 3 to 
repeat a series of panoramic image sensing processes. 
[0219] In this fashion, after the control returns to step 
S304 in Fig. 3 upon canceling (releasing) the second 
shutter switch 124, the next image sensing region is 
selected in step S31 1 . In step S313. the image sensing 
regions which include the previous image sensing 
region and display the sensed images are set in a quick 
review display state and the next image sensing region 
is set in a through display state, so as to display the 
sensed images and an image to be sensed to overlap 
each other, thus allowing the user to easily sense pano- 
ramic images. 

[0220] Figs. 14A and 14B show examples of the 
image sensing window before and after completion of 
sensing of the n-th image in the horizontal panoramic 
mode. Fig. 14A shows the image sensing window 
before sensing the first image, and Fig. 14B shows the 
image sensing window upon canceling the second shut- 
ter switch 124 after the first image is sensed. Although 
not shown, before the second shutter switch 124 is can- 
celed after the first image sensing, the first image sens- 
ing region in Fig. 14A on the image sensing window is 
set in a quick review display state. 
[0221] Figs. 15A and 15B show examples of the 
image sensing window before and after completion of 
sensing of the n-th image in the vertical panoramic 
mode. Fig. 15A shows the image sensing window 
before sensing the first image, and Fig. 15B shows the 
image sensing window upon canceling the second shut- 
ter switch 1 24 after the first image is sensed. Although 
not shown, before the second shutter 124 is canceled 
after the first image sensing, the first image sensing 
region in Fig. 15A on the image sensing window is set in 
a quick review display state. 

[0222] Furthermore. Figs. 12A to 13B show exampler 
of the image sensing window before and after comple- 
tion of sensing of the n-th image. 
[0223] Fig. 12A shows the image sensing window 



before the first image is sensed. Fig. 12B shows the 
image sensing window upon canceling the second shut- 
ter switch 124 after the first image is sensed, i.e., before 
the second image is sensed. 

5 [0224] Although not shown, before the second shutter 
switch 124 is canceled after the first image sensing, the 
first image sensing region in Fig. 12A on the image 
sensing window is set in a quick review display state. 
[0225] Fig. 13A shows the image sensing window 

10 upon canceling the second shutter switch 124 after the 
second image is sensed, i.e., before the third image is 
sensed, and Fig. 13B shows the image sensing window 
upon canceling the second shutter switch 124 after the 
third image is sensed, i.e. before the fourth image is 

75 sensed. 

[0226] Although not shown, before the second shutter 
switch 124 is canceled after the n-th image sensing, the 
n-th image sensing region in Figs. 12A to 13B on the 
image sensing window is set in a quick review display 
20 state. When the second shutter switch 124 is canceled, 
a through display of the (n+1)-th image sensing region 
as the next region is added. 

[0227] Fig. 5 is a flow chart showing the flow of the 
operation of the distance measurement/photometry 

25 process in step S404 shown in Fig. 4 in detail. 

[0228] The system control circuit 118 reads out a 
charge signal from the image sensing element 103. and 
the image processing circuit 106 sequentially reads 
sensed image data via the A/D converter 104 in step 

30 S501 . Using the read image data, the image processing 
circuit 106 makes predetermined computations used in 
the TTL (Through The Lens) AE (auto exposure) proc- 
ess, EF (flash pre-emission) process, and AF (auto 
focus) process. 

35 [0229] Note that the respective processes extract spe- 
cific portions at required positions from ail pixels, and 
use them in computations. In this manner, in the TTL 
AE, EF, and AF processes, optimal computations can 
be made in units of different modes, i.e., a center- 

40 weighted mode, average mode, evaluation mode, and 
the like. 

[0230] The system control circuit 1 1 8 checks using the 
computation results of the image processing circuit 1 06 
in step S502 if the exposure value (AE) is appropriate 

45 (OK). If AE is inappropriate, the system control circuit 
1 18 executes AE control in step S503. The system con- 
trol circuit 118 checks using measurement data 
obtained by the AE control in step S504 if flash emission 
is required. If-fjash emission is not required, the flow 

so returns to step S501 ; otherwise, the system control cir- 
cuit 1 18 sets a flash flag to charge the electronic flash 
1 1 7 in step S505. Then, the flow returns to step S501 . 
[0231] On the other hand, if it is determined in step 
S502 that AE is appropriate, the system control circuit 

55 1 18 stores measurement data and/or setting parame- 
ters in its internal memory or the memory 119. The sys- 
tem control circuit 118 checks using the computation 
results of the image processing circuit 106 and meas- 
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urement data obtained by the AE control in step S506 it 
the white balance (AWB) is appropriate (OK). If the 
white balance is inappropriate, the system control circuit 
118 executes AWB by adjusting the color processing 
parameters using the image processing circuit 106 in $ 
step S507, and the flow then returns to step S501 . 
[0232] On the other hand, if it is determined in step 
S506 that the white balance is appropriate, the system 
control circuit 118 stores measurement data and/or set- 
ting parameters in its internal memory or the memory io 
119. The system control circuit 1 18 checks using meas- 
urement data obtained by the AE control and AWB con- 
trol in step S508 rf a distance measurement result 
indicates in-focus. If the distance measurement result 
does not indicate in-focus. the system control circuit 118 is 
executes AF control in step S509, and the flow returns 
to step S501. On the other hand, il it is determined in 
step S508 that the distance measurement result indi- 
cates in-focus, the system control circuit 118 stores 
measurement data and/or setting parameters in its 20 
internal memory or the memory 119, thus ending this 
processing operation (the processing operation in step 
S404 in Fig. 4). 

[0233] Fig. 6 is a flow chart showing the flow of the 
image sensing process in step S408 in Fig. 4 in detail. 25 
[0234] The system control circuit 118 opens the shut- 
ter 102 having the stop lunction using the exposure con- 
trol means 1 13 in accordance with photometry data 
stored in its internal memory or the memory 1 1 9 in step 
S601, and starts exposure of the image sensing ele- 30 
ment 103 in step S602. The system control circuit 1 18 
checks based on the flash flag in step S603 if the elec- 
tronic flash 1 1 7 is required. If the electronic flash 1 17 is 
required, the system control circuit 118 controls the 
electronic flash 1 17 to emit light in step S604, and the 35 
flow advances to step S605. If it is determined in step 
S603 that the electronic flash 117 is not required, the 
control skips step S604 and advances to step S605. 
[0235] The system control circuit 1 18 checks in step 
S605 if exposure of the image sensing element 103 is 40 
complete in accordance with the photometry data, until 
jt is actually complete. Upon completion of exposure of 
the image sensing element 103, the system control cir- 
cuit 1 18 closes the shutter 102 in step S606, and reads 
a charge signal from the image sensing element 103 45 
and writes sensed image data in the memory 111 via 
the A/D converter 104, image processing circuit 106, 
and memory control circuit 107 or directly from the A/D 
converter 104 via the memory control circuit 107 in step 
S607. Upon completion of a series of processes, this so 
processing operation (the processing operation in step 
S408 in Fig. 4) ends. 

[0236] Fig. 7 is a flow chart showing the flow of the dis- 
play image process in step S409 shown in Fig. 4 in 
detail. 55 
[0237] The system control circuit 118 checks in step 
S701 if a frame process is required in correspondence 
with the selected image sensing mode. If a frame proc- 



ess is required, the system control circuit 118 reads out 
image data written in the memory 1 1 1 and executes 
vertical addition using the memory control circuit 107 
and also the image processing circuit 106 if necessary 
in step S702. Likewise, the system control circuit 118 
executes a color process in step S703. After that, the 
system control circuit 118 writes the processed image 
data in the memory ill, and the flow advances to step 
S704. 

[0238] On the other hand, if it is determined in step 
701 that the frame process is not required, the control 
skips steps S702 and S703. and advances to step 
S704. 

[0239] In step S704, the system control circuit 1 18 
reads out image data from the memory 111, and trans- 
fers display image data to the image display memory 
108 via the memory control circuit 107. 
[0240] Upon completion of a series of processes, this 
processing operation (the processing operation in step 
S409 in Fig. 4) ends. 

[0241] Fig. 8 is a flow chart showing the flow of the 
compression process in step S41 1 in Fig. 4 in detail. 
[0242] In step S801, the system control circuit 118 
executes a pixel squaring process for reading out 
sensed image data written in the memory 111, and con- 
verting the vertical -to-horizontal pixel ratio of the image 
sensing element 103 to 1 : 1 by interpolation using the 
memory control circuit 107 and also the image process- 
ing circuit 106 if necessary. The system control circuit 
118 then writes the processed image data in the mem- 
ory 111. In step S802, the system control circuit 118 
reads out image data written in the memory 1 1 1 and 
executes image compression corresponding to the 
selected mode using the compression/expansion circuit 
112, then ending this processing operation (the 
processing operation in step S41 1 in Fig. 4). 
[0243] Note that the number of types of selectable 
panoramic modes on the panoramic mode selection 
window is 3, i.e., the 2 x 2 panoramic mode, horizontal 
panoramic mode, and vertical panoramic mode in this 
embodiment However, the present invention is not lim- 
ited to these specific modes, and one or a plurality of 
types of an arbitrary number of different panoramic 
modes may be selectable. 

[0244] Also, the display of the panoramic mode selec- 
tion window shown in Fig. 9 is not limited to the one 
exemplified in this embodiment. Alternatively, a display 
with an arbitrary layout or scheme using an image or 
voice may b^e made. 

[0245] On the 2 x 2 panoramic mode image sensing 
window, a total of four images are two-dimensionally 
laid out in a 2 x 2 matrix. However, the present invention 
is not limited to such matrix. For example, an arbitrary 
number of images such as a total of nine images, a total 
of 16 images, and the like may be laid out in a two- 
dimensional or three-dimensional matrix. 
[0246] In the above embodiment, on the horizontal 
and vertical panoramic mode image sensing windows. 
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two images are continuously displayed. However, the 
present invention is not limited to such specific display, 
and an arbitrary number of images (e.g.. three images, 
four images, and the like) may be one-dimensionally 
displayed. 

[0247J The displays of the panoramic mode image 
sensing windows shown in Figs. 10A to 15B are not lim- 
ited to these examples, and displays with arbitrary lay- 
outs and schemes using images, voices, characters, 
symbols, and the like may be made. 10 
[0248] in the above-mentioned embodiment, in the 
horizontal and vertical panoramic modes, the image 
sensing direction is inhibited from being reversed after 
the first image is sensed. However, the present inven- 
tion is not limited to this, and the image sensing direc- is 
tion may be reversed as needed. Also, the image 
sensing direction may be changed in one of arbitrary 
directions including the right and left directions, up and 
down directions, or oblique directions. In the present 
invention, when a large number of image display so 
regions can be simultaneously displayed by increasing 
the display area of the image display unit 1 1 0. the image 
sensing direction can be changed in one of arbitrary 
directions including the right and left directions, up and 
down directions, or oblique directions. 25 
[0249] In the above embodiment, image sensing can 
be redone for only the immediately preceding sensed 
image in the horizontal and vertical panoramic modes. 
However, the present invention is not limited to this. For 
example, image sensing may be redone for an image 30 
sensed an arbitrary number of images before. 
[0250] In the above embodiment; the next image dis- 
play region for displaying an image to be sensed is set 
in the through display state, and that for displaying the 
already sensed image is set in the quick review display 35 
state, to display the playback image of the already 
sensed image and the next image to be sensed so that 
their boundary regions partially overlap each other 
However, the present invention is not limited to such 
specific scheme. For example, the images may be dis- <o 
played without making their boundary regions overlap 
each .other. 

[0251] In the explanatory views of the panoramic 
modes in Figs. 10A to 15B. for example, comment 
words "LIVE" and "FREEZE" are added within the 45 
image display regions. However, the actual image dis- 
play unit 110 may or may not display these comment 
words. 

[0252] Note that the recording media 200a and 200b 
are not limited to memory cards such as PCMCIA so 
cards, compact flash cards, or the like, hard disks, and 
the like, but may use micro DATs (digital audio tapes), 
magnetooptical disks, optical disks such as CD-Rs, CD- 
WRs. or the like, phase change optical disks such as 
DVDs, and the like. 55 
[0253] Also, the recording media 200a and 200b may 
use hybrid media that integrate memory cards, hard 
disks, and the like. Furthermore, such hybrid media may 



include detachable media. 

[0254] In the description of the embodiment, the 
recording media 200a and 200b are independent from 
the image processing apparatus 100 and are arbitrarily 
connectable. One or both the recording media 200a and 
200b may be permanently connected to the image 
processing apparatus 100. 

[0255] An arbitrary number (one or a plurality) of 
image recording media 200a or 200b may be connecta- 
ble to the image processing apparatus 100. 
[0256] In the above description, the recording media 
200a and 200b are attached to the image processing 
apparatus. However, one or a combination of a plurality 
of recording media may be used. 
[0257] As described in detail above, according to the 
image processing method and apparatus of the first 
embodiment, the image sensing purposes are not lim- 
ited, which is very convenient. 

[0258] Also, according to the storage medium of the 
first embodiment, the aforementioned image processing 
apparatus of the first embodiment can be smoothly con- 
trolled. 

(Second Embodiment) 

[0259] Fig. 16 shows the arrangement of an image 
processing apparatus according to the second embodi- 
ment of the present invention. 

[0260] Referring to Fig. 16. reference numeral 400 
denotes an image processing apparatus. 
[0261 ] Reference numeral 1 4 denotes an image sens- 
ing element for converting an optical image into an elec- 
trical signal; and 16, an A/D converter for converting an 
analog signal output from the image sensing element 14 
into a digital signal. 

[0262] Reference numeral 18 denotes a timing gener- 
ation circuit for supplying clock signals and control sig- 
nals to the image sensing element 14, the A/D converter 
16, and a D/A converter 26. The timing generation cir- 
cuit 18 is controlled by a memory control circuit 22 and 
system control circuit 50. 

[0263] Reference numeral 20 denotes an image 
processing circuit which performs predetermined pixel 
interpolation and color conversion for data supplied 
from the A/D converter 16 or the memory control circuit 
22. 

[0264] The image processing circuit 20 makes a pre- 
determined computation using sensed image data to 
execute a TTL- (Through The Lens) AF (auto focus) 
process, AE (auto exposure) process, and EF (flash 
pre-emission) process in which the system control cir- 
cuit 50 controls an exposure control means 340 and dis- 
tance measurement control means 342 on the basis of 
the obtained computation result. 
[0265] Furthermore, the image processing circuit 20 
makes a predetermined computation using sensed 
image data, and executes a TTL AWB (auto white bal- 
ance) process on the basis of the obtained computation 
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result. 

[0266] Reference numeral 22 denotes the memory 
control circuit, which controls the A/D converter 16. the 
timing generation circuit 18, the image processing cir- 
cuit 20, an image display memory 24, the D/A converter 
26. a memory 30, and a compression/expansion circuit 
32 

10267] Data output from the A/D converter 16 is writ- 
ten in the image display memory 24 or memory 30 via 
the image processing circuit 20 and memory control cir- 
cuit 22 or directly via the memory control circuit 22. 
[0268] Reference numeral 24 denotes the image dis- 
play memory; 26, the D/A converter; and 28, an image 
display unit comprising a TFT LCD or the like. Display 
image data written in the image display memory 24 is 
displayed by the image display unit 28 via the D/A con- 
verter 26. 

[0269] When sensed image data are displayed at 
image sensing timings using the image display unit 28, 
an electronic viewfinder function can be implemented. 
[0270] The image display unit 28 can arbitrarily turn 
on/off rts display in response to an instruction from the 
system control circuit 50. When the display is turned off, 
consumption power of the image processing apparatus 
400 can be greatly reduced. 

[0271 ] Reference numeral 30 denotes the memory for 
storing a still image and moving image. The memory 30 
has a storage capacity large enough to store a predeter- 
mined number of still images or a moving image for a 
predetermined period of time. With this arrangement, 
even in a continuous or panoramic image sensing mode 
for continuously sensing a plurality of still images, many 
images can be written in the memory 30 at high speed. 
Also, the memory 30 can be used as a work area of the 
system control circuit 50. 

[0272] Reference numeral 32 denotes the compres- 
sion/expansion circuit for compressing/expanding 
image data by adaptive discrete cosine transformation 
(ADCT) or the like. The compression/expansion circuit 
32 reads an image stored in the memory 30, com- 
presses or expands it, and writes the processed data in 
the memory 30. 

[0273] Reference numeral 48 denotes an electronic 
flash which has a function of projecting AF auxiliary light 
and the flash light control function. 
[0274] The exposure control means 340 and distance 
measurement control means 342 are controlled using 
TTL, and the system control circuit 50 controls the expo- 
sure control means 340 and distance measurement 
control means 342 on the basis of the computation 
result of the image processing circuit 20 using sensed 
image data. 

[0275] Reference numeral 50 denotes the system 
control circuit for controlling the overall image process- 
ing apparatus 400; and 52, a memory for storing con- 
stants, variables, programs, and the like for operations 
of the system control circuit 50. 
[0276] Reference numeral 54 denotes an indication 



unit which includes a liquid crystal display device, loud- 
speaker, and the like, and indicates operation states, 
messages, and the like using characters, images, 
voices, and the like in accordance with execution of a 

5 program by the system control circuit 50. The indication 
unit 54 is placed at one or a plurality of easy-to-see 
positions around the console of the image processing 
apparatus 400. and is comprised of a combination of an 
LCD, LEDs, tone generation element, and the like. 

10 [0277] Some functions of the indication unit 54 are 
placed within an optical viewfinder 404. 
[0278] Of the indication contents of the indication unit 
54, those indicated on an LCD or the like include, e.g., a 
single shot/continuous shot indication, self-timer indica- 

15 tion, compression ratio indication, recording pixel count 
indication, recording image count indication, remaining 
photographable image count indication, shutter speed 
indication, aperture value indication, exposure correc- 
tion indication, flash indication, red-eye suppression 

20 indication, macro-image sensing indication, buzzer set- 
ting indication, timepiece battery remaining capacity 
indication, error indication, information indication using 
a plurality of digits of numerals, attached/detached state 
indication of recording media 500 and 510. 

25 attached/detached state indication of a lens unit 300, 
communication l/F operation indication, date/time indi- 
cation, indication that indicates a connection state with 
an external computer, and the like. 
[0279] Of the indication contents of the indication unit 

30 54, those indicated within the optical viewfinder 404 
include, e.g., an in-focus indication, image sensing 
ready indication, camera shake alert indication, flash 
charging indication, flash charging completion indica- 
tion, shutter speed indication, aperture value indication. 

35 exposure correction indication, recording medium write 
access indication, and the like. 

[0280] Furthermore, of the indication contents of the 
indication unit 54, those indicated by LEDs or the like 
include, e.g., in-focus indication, image sensing ready 

40 indication, camera shake alert indication, flash charging 
indication, flash charging completion indication, record- 
ing medium write access indication, macro-image sens- 
ing setting notification indication, secondary battery 
charged state indication, and the like. 

45 [0281 ] Of the indication contents of the indication unit 
54, those to be indicated by lamps include, e.g., a self- 
timer notification lamp, and the like. The self-timer noti- 
fication lamp may be commonly used as that for emitting 
AF auxiliary light. 

so [0282] Reference numeral 56 denotes an electrically 
erasable/recordable nonvolatile memory, which uses, 
e.g.. an EEPROM. 

[0283] Reference numerals 60. 62. 64, 66. 68. 70, 72. 
74. and 76 denote operation means for inputting various 
55 operation instructions of the system control circuit 50. 
These operation means are constructed by one or a plu- 
rality of combinations of a switch, a dial, a touch panel, 
a pointer using line of sight detection, a voice recogni- 
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tion device, and the like. 

[0284] These operation means will be explained in 
detail below. 

[0285] Reference numeral 60 denotes a mode dial 
switch, which can switch various function modes such s 
as power OFF, an automatic image sensing mode, 
image sensing mode, panoramic image sensing mode 
playback mode, multi-frame playback/erase mode, PC 
connection mode, and the like. 

[0286] Reference numeral 62 denotes a shutter switch 10 
SWl, which is turned on in the middle of operation of a 
shutter button (not shown), and instructs start of an AF 
(auto focus) process, AE (auto exposure) process, AWB 
(auto white balance) process. EF (flash pre-emission) 
process, and the like. 75 
[0287] Reference numeral 64 denotes a shutter switch 
SW2, which is turned on upon completion of operation 
of the shutter button (not shown), and instructs start of a 
series of processes including an exposure process for 
writing a signal read out from the image sensing ele- so 
ment 14 as image data in the memory 30 via the A/D 
converter 16 and memory control circuit 22. a develop- 
ment process using computation results in the image 
processing circuit 20 and memory control circuit 22, and 
a recording process for reading out image data from the 25 
memory 30. compressing the readout data by the com- 
pression/expansion circuit 32. and writing the com- 
pressed image data in the recording medium 500 or 
510. 

[0288] Reference numeral 66 denotes a sefec- 30 
tion/change switch which can select and change vari- 
ous functions upon executing image sensing and 
playback in a panoramic mode or the like. 
[0289] Reference numeral 68 denotes a determina- 
tion/execution switch which can determine and execute 35 
various functions upon executing image sensing and 
playback in a panoramic mode or the like. 
[0290] Reference numeral 70 denotes a console 
including various buttons, touch panel, and the like, 
which include a menu button, set button, macro button. 40 
multi-frame playback new page button, flash setting but- 
ton,, single shot/continuous shot/self-timer switch but- 
ton, menu movement + (plus) button, menu movement - 
(minus) button, playback image movement + (plus) but- 
ton, playback image movement - (minus) button, sensed 45 
image quality selection button, exposure correction but- 
ton, date/time setting button, image display ON/OFF 
button for turning on/off the image display unit 28, quick 
review ON/OFF button for setting a quick review func- 
tion of automatically playing back sensed image data on so 
the image display unit 28 immediately after image sens- 
ing, and the like. 

[0291 ] Reference numeral 72 denotes a compression 
mode switch, which is used for selecting a compression 
ratio of JPEG compression or selecting a CCDRAW 55 
mode for directly recording a signal output from the 
image sensing element and converted into a digital sig- 
nal on a recording medium. 



[0292] The JPEG compression mode includes, e.g.. a 
normal mode and fine mode. 

[0293] In the JPEG compression mode, image data, 
which is read out from the image sensing element 14 
and is written in the memory 30 via the A/D converter 
16, image processing circuit 20, and memory control 
circuit 22. is read out, and is compressed at the selected 
compression ratio by the compression/expansion circuit 
32. After that, the compressed image data is recorded 
on the recording medium 500 or 510. 
[0294] In the CCDRAW mode, image data, which is 
directly read out in units of lines in correspondence with 
the pixel formats of color filters of the image sensing 
element 14, and is written in the memory 30 via the A/D 
converter 16 and memory control circuit 22. is read out, 
and is recorded on the recording medium 500 or 510. 
[0295] Reference numerals 74 and 76 denote zoom 
switches which 200m a photographing lens 310 via a 
zoom control means 344 of the lens unit 300. Note that 
the switch 74 is a zoom switch WIDE used for zooming 
the lens in the wide-angle direction, and the switch 76 is 
a zoom switch TELE used for zooming the lens in the 
telephoto direction. The switches 74 and 76 may be 
integrated. 

[0296] Reference numeral 80 denotes a power supply 
control means, which is comprised of a battery detec- 
tion circuit, a DC-DC converter, a switch circuit for 
switching a block to be energized, and the like. The 
power supply control means 80 detects the pres- 
ence/absence, type, and remaining battery amount of a 
battery attached, controls the DC-DC converter on the 
basis of such detection results and an instruction from 
the system control circuit 50. and supplies a required 
voltage to the respective units including the recording 
medium for a required period of time. 
[0297] Reference numerals 82 and 84 denote connec- 
tors; and 86, a power supply means including a primary 
battery such as an alkali battery, lithium battery, or the 
like, a secondary battery such as an NiCd battery, NiMH 
battery, Li Battery, or the like, an AC adapter, and the 
like. 

[0298] Reference numerals 90 and 94 denote inter- 
faces with recording mecfia such as a memory card, 
hard disk, and the like; 92 and 96, connectors for con- 
necting the recording media such as a memory card, 
hard disk, and the like; and 98, a recording medium 
attachment/detachment detection means for detecting 
whether or not the recording medium 500 or 510 is 
attached to ther connector 92 and/or the connector 96. 
[0299] Note that this embodiment has two sets of 
interfaces and connectors that receive the recording 
media. Of course, the number of sets of interfaces and 
connectors that receive the recording media is not par- 
ticularly limited. Also, combinations of interfaces and 
connectors of different standards may be used. 
[0300] As the interface and connector, those comply- 
ing with the standards of a PCMCIA card. CF (compact 
flash) card, and the like may be used. 
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[0301] Furthermore, when the interfaces 90 and 94, 
and connectors 92 and 96 use those complying with the 
standards of a PCMCIA card, CF (compact flash) card, 
and the like, various communication cards such as a 
LAN card, modem card, USB card, IEEE 1394 card, 5 
P1284 card, SCSI card. PHS, and the like are con- 
nected thereto, image data and associated manage- 
ment information can be transferred between the image 
processing apparatus 400 and an external computer or 
its peripheral devices such as a printer and the like. w 
[0302] Reference numeral 404 denotes the optical 
viewfinder, which allows image sensing using the opti- 
cal viewfinder alone without using the electronic view- 
finder function implemented by the image display unit 
28. In the optical viewfinder 404, some functions of the is 
indication unit 54, e.g., an in-focus indication, camera 
shake alert indication, flash charging indication, shutter 
speed indication, aperture value indication, exposure 
correction indication, and the like are placed. 
[0303] Reference numeral 410 denotes a communica- 20 
tion means having various communication functions 
such as RS232C, USB, IEEE 1394. P1284, SCSI, 
modem, LAN, radio communication, and the like. 
[0304] Reference numeral 41 2 denotes a connector or 
antenna (in case of a radio communication), which con- 25 
nects the image processing apparatus 400 to another 
device using the communication means 410. 
[0305] Reference numeral 420 denotes an interlace 
for connecting the image processing apparatus 400 to 
the lens unit 300 in a lens mount 406; 422, a connector 30 
for electrically connecting the image processing appara- 
tus 400 to the lens unit 300; and 424, a lens attach- 
ment/detachment detection means for detecting 
whether or not the lens unit 300 is attached to the lens 
mount 406 and/or the connector 422. 35 
[0306] The connector 422 transfers control signals, 
status signals, data signals, and the like between the 
image processing apparatus 400 and lens unit 300. and 
also has a function of supplying currents of various volt- 
ages. Also, the connector 422 may have an arrange-' 40 
ment that realizes optical communications, voice 
communications, or the like in place of electrical com- 
munications. 

[0307] Reference numeral 500 denotes the recording 
medium such as a memory card, hard disk, or the like. 45 
The recording medium 500 comprises a recording unit 
502 comprised of a semiconductor memory, magnetic 
disk, or the like, an interface 504 with the image 
processing apparatus 400, and a connector 506 for con- 
necting the image processing apparatus 400. so 
[0308] Reference numeral 510 denotes the recording 
medium such as a memory card, hard disk, or the like. 
The recording medium 510 comprises a recording unit 
512 comprised of a semiconductor memory, magnetic 
disk, or the like, an interface 514 with the image ss 
processing apparatus 400, and a connector 51 6 for con- 
necting the image processing apparatus 400. 
[0309] Reference numeral 300 denotes the exchange- 



able lens type lens unit. 

[0310] Reference numeral 306 denotes a lens mount 
for mechanically coupling the lens unit 300 to the image 
processing apparatus 400. The lens mount 306 
includes various functions of electrically connecting the 
lens unit 300 to the image processing apparatus 400. 
[0311] Reference numeral 310 denotes the photo- 
graphing lens; and 312, a shutter with a stop function. 
[0312] Reference numeral 320 denotes an interface 
for connecting the lens unit 300 to the image processing 
apparatus 400 in the lens mount 306; and 322, a con- 
nector for electrically connecting the lens unit 300 to the 
image processing apparatus 400. 
[0313] The connector 322 transfers control signals, 
status signals, data signals, and the like between the 
image processing apparatus 400 and lens unit 300, and 
also has a function of supplying currents of various volt- 
ages. Also, the connector 322 may have an arrange- 
ment that realizes optical communications, voice 
communications, or the like in place of electrical com- 
munications. 

[0314] Reference numeral 340 denotes the exposure 
control means for controlling the shutter 312 with the 
stop function. The exposure control means 340 also has 
a flash light control function in cooperation with an elec- 
tronic flash 48. 

[0315] Reference numeral 342 denotes the distance 
measurement control means for controlling focusing of 
the photographing lens 310; and 344, the zoom control 
means for controlling zooming of the photographing 
lens 310. 

[031 6] Reference numeral 350 denotes a lens system 
control circuit for controlling the overall lens unit 300. 
The lens system control circuit 350 has a function of a 
memory for storing constants, variables, programs, and 
the like for operations, and a function of a nonvolatile 
memory for holding identification information such as a 
number unique to the lens unit 300, management infor- 
mation, function information such as a full -open aper- 
ture value, minimum aperture value, focal length, and 
the like, the current and previous setting values, and the 
like. 

[0317] The operation of the second embodiment will 
be explained below with reference to Figs. 17, 18, 19, 
20, and 21, and Figs. 5. 6. 7, and 8 common to the first 
embodiment. 

[0318] Figs. 17. 18, 19, 20, and 21 are flow charts 
showing the main routine of the image processing appa- 
ratus 400 of this embodiment. 

[031 9] The operation of the image processing appara- 
tus 400 will be described with the aid of Figs. 17. 18, 19, 
20, and 21. 

[0320] Upon power ON after battery exchange or the 
like, the system control circuit 50 initializes flags, control 
variables, and the like, and initializes the respective 
units of the image processing apparatus 400 (step 
S101). 

[0321 ] The system control circuit 50 checks the setting 
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position of the mode dial 60. If the mode dial 60 is set at 
a power OFF position (step S102), the system control 
circuit 50 executes a predetermined end process (step 
S103» More specifically, the system control circuit 50 
cnanges the indications of the respective indication 
umts to an end state, protects an image sensing unit by 
dosing a barrier if the lens unit 300 comprises the bar- 
rier or the like serving as a protection means, records 
required parameters and setting values including flags, 
control variables, and the like, and the setting mode in 
the nonvolatile memory 56, cuts off unnecessary power 
supply to the respective units of the image processing 
apparatus 400 including the image display unit 28 by the 
power supply control means 80, and so forth. After that, 
the flow returns to step $102. 

[0322] If the mode dial 60 is set at any of other mode 
positions (step S102), the system control circuit 50 
eneefcs using the power supply control means 80 if the 
remaining capacity and operation state of the power 
suoply means 86 comprising batteries and the like pose 
any problem m the operation of the image processing 
apparatus 400 (step S104). If any problem is found, a 
predetermined alert indication is made by means of an 
imagp or voice using the indication unit 54 (step S106), 
and the flow then returns to step S102. 
[0323] When the image display of the image display 
unit 23 is ON, a predetermined alert display is made by 
means of an image or voice also using the image dis- 
play unit 28. 

[0324] If no problem is found in the power supply 
means 86 (step S104), the system control circuit 50 
checks if the operation state of the recording medium 
500 or 510 poses any problem in the operation of the 
image processing apparatus 400, in particular, record- 
ing/playback of image data to/from the recording 
medium (step S105). If any problem is found, a prede- 
termined alert indication is made by means of an image 
or voice using the indication unit 54 (step S106), and the 
flow then returns to step Si 02. 
[0325] When the image display of the image display 
unit 28 is ON, a predetermined alert display is made by 
means of an image or voice also using the image dis- 
play unit 28. 

[0326] If no problem is found in the operation state of 
the recording medium 500 or 510 (step S105), various 
setting states of the image display apparatus 400 are 
indicated by means of an image or voice using the indi- 
cation unit 54 (step S107). When the image display of 
the image display unit 28 is ON, various setting states of 
the image display apparatus 400 are indicated by 
means of an image or voice also using the image dis- 
play unit 28. 

[0327] On the other hand, if the mode dial 60 is set at 
an image sensing mode position (step S108). a prede- 
termined image sensing process is executed (step 
S109). Upon completion of the process, the flow returns 
to step S102. 

[0328] If the mode dial 60 is set at any of other mode 



positions (step S108), the system control circuit 50 exe- 
cutes a process corresponding to the selected mode 
(step S110). Upon completion of the process, the flow 
returns to step S102. 

5 [0329] If the mode dial 60 is set at a panoramic mode 
position (step S108), the system control circuit 50 
checks if the zoom position of the photographing lens 
310 is a predetermined home position (step S121). If 
the photographing lens is not located at the home posi- 

io tion, the system control circuit 50 moves the zoom posi- 
tion to the home position (step Si 22). 
[0330] Also, when the photographing lens 310 uses a 
selectable lens that can select one of a plurality of differ- 
ent focal lengths, the system control circuit 50 checks if 

75 that selectable lens of the photographing lens 310 is 
located on the predetermined home position side (step 
S121). If the selectable lens is not located on the home 
position side, the system control circuit 50 moves the 
selectable lens to the home position side (step S122). 

20 [0331 ] As an example of the home position, in case of 
a zoom lens, the WIDE end as a focal length position on 
the wide-angle side suitable for panoramic image sens- 
ing can be used. 

[0332] In case of a focal length selectable lens, a lens 
25 with a focal length on the WIDE side as that on the wide- 
angle end suitable for panoramic image sensing may be 
selected. 

[0333] The present invention is not limited to such 
specific example, and any focal length may be set at a 
30 home position as long as it is suitable for panoramic 
image sensing. 

[0334] The system control circuit 50 displays a pano- 
ramic mode selection window on the image display unit 
(step S1 11). 

35 [0335] The user of the image processing apparatus 
400 can select an arbitrary one of a plurality of pano- 
ramic image sensing modes by observing the pano- 
ramic mode selection window displayed on the image 
display unit 28. 

40 [0336] The panoramic image sensing modes include 
a 2 x 2 panoramic mode, horizontal panoramic mode, 
vertical panoramic mode, and the like. In these modes, 
by combining a plurality of sensed images, a panoramic 
image, which is extended two-dimensionally, horizon- 

45 tally, or vertically, can be obtained. 

[0337] A series of sensed images are appended with 
information indicating their positional relationship, and 
can be combined to generate a signal panoramic image 
by the image* processing apparatus 400 itself or after 

so they are transferred to, e.g., a computer or the like. 

[0338] Note that an example of the panoramic mode 
selection window is the same as that shown in Fig. 9 
that has already been explained in the first embodiment. 
[0339] Referring to Fig. 9, the left icon is used for 

55 selecting the 2 x 2 panoramic mode for sensing a two- 
dimensional panoramic image, the central icon is used 
for selecting the horizontal panoramic mode for sensing 
a horizontal panoramic image, and the right icon is used 
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for selecting a vertical panoramic mode for sensing a 
vertical panoramic image. 

(0340] The user of the image processing apparatus 
400 can select a desired panoramic mode by the selec- 
tion/change switch 66 (e.g., [+] key), and can determine s 
that panoramic mode by the determination/execution 
switch 68 (e.g., [SET] key). 

[0341] When the panoramic mode is selected using 
the selection/change switch 66 and determination/exe- 
cution switch 68, the system control circuit 50 sets initial io 
values and memory areas of flags, parameters, and var- 
iables, reads out a display window serving as a user 
interface, and so forth in a 2 x 2 panoramic mode initial 
setup process (step Si 13). a vertical panoramic mode 
initial setup process (step S1 14), or a horizontal pano- is 
ramie mode initial setup process (step Si 15) in corre- 
spondence with the selected panoramic mode (step 
S112), and displays a panoramic image sensing win- 
dow corresponding to the selected mode on the image 
display unit 28 (step S1 16). The flow then advances to 20 
step S131. 

[0342] Note that Figs. 10A and 10B show examples of 
the horizontal panoramic mode image sensing window, 
Figs. 1 1 A and 1 1 B show examples of the vertical pano- 
ramic mode image sensing window, and Figs. 12A and 25 
1 2B show examples of the 2 x 2 panoramic mode image 
sensing window, as in the description of the first embod- 
iment. 

[0343] Fig. 10 A shows an image sensing window that 
creates a panoramic window by continuously repeating 30 
image sensing to the right and Fig. 10B shows an 
image sensing window that creates a panoramic win- 
dow by continuously repeating image sensing to the left. 
[0344] Referring to Figs. 10A and 10B, a through 
image from the electronic viewfinder is displayed on 35 
region "1" to sense the first image of a series of pano- 
ramic images. Before completion of image sensing of 
the first image, rightward image sensing (Fig. 10A) or 
leftward image sensing (Fig. 10B) can be desirably 
selected using the selection/change switch 66 (e.g., [+] 40 
key). 

[0345] The image sensing direction is determined by 
sensing the first image by pressing the shutter button in 
desired one of the display window states shown in Figs. 
1 0A and 1 0B. 45 
[0346] Note that the user can end image sensing in 
the horizontal panoramic mode by pressing the determi- 
nation/execution switch 68 (e.g., [SET] key). 
[0347] Fig. 1 1 A shows an image sensing window that 
creates a panoramic window by continuously repeating so 
image sensing upward, and Fig. 11B shows an image 
sensing window that creates a panoramic window by 
continuously repeating image sensing downward. 
[0348] Referring to Figs. 11A and 11B, a through 
image from the electronic viewfinder is displayed on ss 
region "1 " to sense the first image of a series of pano- 
ramic images. Before completion of image sensing of 
the first image, upward image sensing (Fig. 11 A) or 



downward image sensing (Fig. 11B) can be desirably 
selected using the selection/change switch 66 (e.g.. [+] 
key). 

[0349] The image sensing direction is determined by 
sensing the first image by pressing the shutter button in 
desired one of the display window states shown in Figs. 
HAand11B. 

[0350] Note that the user can end image sensing in 
the vertical panoramic mode by pressing the determina- 
tion/execution switch 68 (e.g., [SET] key). 
[0351] Fig. 12A shows an image sensing window for 
the first image upon creating a 2 x 2 panoramic window 
by two-dimensionally repeating image sensing of a total 
of four images in the vertical and horizontal directions, 
and Fig. 12B shows an image sensing window for the 

second image. — 

[0352] Referring to Figs. 12A and 12B. a through 
image from the electronic viewfinder is displayed on, 
e.g., an upper right region to sense the first one of pan- 
oramic images. 

[0353] Upon completion of image sensing of the first 
image, the playback image of the sensed image is dis- 
played on, e.g., the upper right region, and a through 
image from the electronic viewfinder is displayed on, 
e.g., the lower right region to sense the second image. 
[0354] The image sensing region of the electronic 
viewfinder is switched in turn to the next region after 
each image sensing. However, image sensing may be 
started from an arbitrary region, or may be redone by 
returning to the already sensed region. For this pur- 
pose, the image sensing region can be desirably shifted 
using the selection/change switch 66 (e.g., [+] switch). 
[0355] The image sensing region to be re-sensed is 
determined and image sensing is executed by pressing 
the shutter button in a display window state that selects 
a desired region. 

[0356] Note that the user can end image sensing in 
the 2 x 2 panoramic mode by pressing the determina- 
tion/execution switch 68 (e.g.. [SET] key). 
[0357] The system control circuit 50 initializes flags 
and variables set in its internal memory or the memory 
52. For example, the system control circuit 50 resets a 
panoramic image sensing start flag (step S131), sets 
variables m and n, and initializes them to m = 1 and n = 
1, respectively (steps S133 and S132). 
[0358] Note that the panoramic image sensing start 
flag is reset to indicate that no panoramic image sens- 
ing processes have been done. 

[0359] Also^ the variable n indicates the position of the 
current image sensing region in image sensing of each 
panoramic mode. 

[0360] The variable m is used for returning the image 
sensing region of interest to the immediately preceding 
region or canceling such return, upon executing image 
sensing again in the horizontal and vertical panoramic 
modes. 

[0361] An example of the method of using these flag 
and variables will be explained later. 
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[0362] K the mode dial 60 is set at any of mode posi- 
tions other than the panoramic mode (step S134). the 
system control circuit 50 ends image sensing in the pan- 
oramic mode, and the flow returns to step S102. In this 
manner, the user of the image processing apparatus 
400 can end the panoramic mode anytime he or she 
wants, and can select another mode. 
[0363] If end of the panoramic mode is selected by 
pressing the determination/execution switch 68 (e.g., 
[SET] key) (step S135) while the mode dial 60 is kept 
set at the panoramic mode position (step Si 34), i.e.. the 
display window of the image display unit 28 keeps dis- 
playing the corresponding panoramic image sensing 
window, a panoramic mode end confirmation message 
is displayed. Furthermore, if confirmation of end is 
sel cted (step S136), the flow returns to step Si 11 of 
displaying the panoramic mode selection window. 
[0364] If the user has pressed the 200m switch WIDE 
74 or zoom switch TELE 76 (step S161). the system 
control circuit 50 checks the status of the panoramic 
image sensing start flag (step S162). 
[0365] If the panoramic image sensing start flag is 
reset, i.e., if the first image is not sensed yet in a series 
of panoramic image sensing processes (step Si 62). the 
system control circuit 50 executes predetermined zoom 
operation for zooming the photographing lens 310 via 
the interface 420, connector 422. connector 322. inter- 
face 320, lens system control circuit 350, and zoom 
control means 344 in accordance with an instruction 
input by the zoom switch WIDE 74 or zoom switch TELE 
76 (step S163). The flow then advances to step S137. 
[0366] If the panoramic image sensing start flag is set. 
i.e., if the first image has already been sensed in a 
series of panoramic image sensing processes (step 
Si 62), the system control circuit 50 makes a predeter- 
mined alert display by means of an image or voice using 
the indication unit 54 and/or the image display unit 28 
(step S164). The flow advances to step S137 without 
executing any zoom operation. 

[0367] With this control, an image which is unsuitable 
for panoramic synthesis can be prevented from being 
sensed by zoom operation of the photographing lens 
310 during a series of panoramic image sensing proc- 
esses. 

[0368] If the selection/change switch 66 (e.g., [+] key) 
has been pressed while the image display unit 28 dis- 
plays one of the above-mentioned panoramic image 
sensing windows (step $137), the system control circuit 
50 executes a predetermined process in correspond- 
ence with the setups of the panoramic mode (step 
S138). 

[0369] If the selectiorVchange switch 66 (e.g., [+] key) 
is not pressed (step S137), the flow advances to step 
S146. 

[0370] If the 2 x 2 panoramic mode is selected (step 
S138), the system control circuit 50 sets the variable n 
to be n = n + 1 to select the next image sensing region 
(step S139). If n = 5 (step S140). the first image sensing 



region is selected by setting n = n - 4 (step S141). After 
that, the flow advances to step Si 46. 
[0371] More specifically, every time the selec- 
tion/change switch 66 (e.g., [+] key) is pressed, the 
5 image sensing region of interest shifts to one of the four 
image sensing regions in the 2 x 2 panoramic mode. For 
example, the image sensing region of interest shifts in 
the order of the upper right region, lower right region, 
lower left region, upper left region, upper right region, 

10 lower right region 

[0372] On the other hand, if the horizontal or vertical 
panoramic mode is selected (step Si 38), and if the pan- 
oramic image sensing start flag is reset, i.e., if the first 
image is not sensed yet in a series of panoramic image 
is sensing processes (step S142), the system control cir- 
cuit 50 reverses the image sensing direction of the 
selected panoramic mode (step S143). After that, the 
flow advances to step S146. 

[0373] Note that the image sensing direction can be 
20 repetitively reversed before the first image is sensed. 
[0374] Figs. 10A and 10B that have already been 
described show examples of an image sensing direction 
reverse window in the horizontal panoramic mode. 
[0375] Fig. 10A shows a window upon repeating 
25 image sensing to the right, and Fig. 10B shows a win- 
dow upon repeating image sensing to the left. 
[0376] Figs. 11A and 11B that have already been 
described show examples of an image sensing direction 
reverse window in the vertical panoramic mode. 
30 [0377] Fig. 11 A shows a window upon repeating 
image sensing upward, and Fig. 11B shows a window 
upon repeating image sensing downward. 
[0378] On the other hand, if the horizontal or vertical 
panoramic mode is selected (step S138) and if the pan- 
35 oramic image sensing start flag is set, i.e.. if the first 
image has already been sensed in a series of pano- 
ramic image sensing processes (step S142), the sys- 
tem control circuit 50 computes m = m x (-1 ), the sign of 
which changes like -1, +1, -1, +1,... in every computa- 
40 tion (step Si 44), and also computes n = n + m for the 
variable n, which changes like n = n-1,n = n+1,n = n 
- 1 , n m n + 1 .... in every computation (step Si 45). After 
that, the flow advances to step S146. 
[0379] With this control, movement to the already 
45 sensed region (which is limited to the one sensed last) 
or cancel that movement can be repetitively selected to 
sense an image again. 

[0380] After panoramic image sensing of the n-th 
image region, the user presses the selection/change 

so switch 66 (e.g., [+] key) to confirm if quick review images 
of the (n-l)-th and n-th image regions smoothly join at 
their boundary regions. Then, the user temporarily 
releases the selection/change switch 66 (e.g.. [+] key) to 
display a through image on the n-th image region, and 

55 presses the selection/change switch 66 (e.g.. [+] key) 
once again to display the original quick review image on 
the n-th image region and a through image on the (n+1 )- 
th image region, thus executing the next panoramic 
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image sensing. 

[0381] After panoramic image sensing of the n-th 
image region, upon pressing the selection/change 
switch 66 (e.g., [+] key) to confirm if quick review images 
of the (n-1)-th and n-th image regions smoothly join at 
their boundary regions, if the user determines that 
image sensing must be redone, he or she temporarily 
releases the selection/change switch 66 (e.g., [+] key) to 
display a through image on the n-th image region, and 
presses the shutter switch SW2 in that window state to 
replace the image with the newly sensed image. Then, 
the user can execute the next panoramic image sens- 
ing. 

[0382] The system control circuit 50 selects the n-th 
image region (step S146). If the window layout must be 
changed in correspondence with the selected image 
region (step S147), the system control circuit 50 
switches and displays a new window layout (step S148), 
and the flow then advances to step S1 71 . 
[0383] In this fashion, a window corresponding to the 
reversed image sensing direction, as described above, 
a window corresponding to the changed image sensing 
region, a window that has updated characters and 
images for comments, or the like can be displayed as 
needed. 

[0384] Upon sensing the second and subsequent 
images, the image sensing regions that display already 
sensed images are set in a quick review display state, 
and the image sensing region selected for the next 
image sensing is set in a through display state, thus dis- 
playing a playback image of the already sensed image 
and the next image to be sensed to partially overlap 
each other. In this manner, the user of the image 
processing apparatus 400 can easily determine the next 
image sensing angle. 

[0385] The system control circuit 50 sets a through 
-display state in which sensed image data is displayed 
as it is sensed (step S171), and the flow advances to 
step S172. 

[0386] In through display, data, which have been writ- 
ten in the image display memory 24 via the image sens- 
ing element 14, A/D converter 16, image processing 
circuit 20, and memory control circuit 22, are displayed 
on the aforementioned image sensing region of the 
image display unit 28 via the memory control circuit 22 
and D/A converter 26, thereby implementing an elec- 
tronic viewfinder function. 

[0387] If the shutter switch SW1 is not pressed (step 
S1 72), the flow returns to step S134. 
[0388] If the shutter switch SW1 has been pressed 
(step S172). the system control circuit 50 sets the dis- 
play state of the image display unit 28 in a freeze display 
state (step S1 73). and the flow advances to step S1 74. 
[0389] In freeze display, image data on the image dis- 
play memory 24 is inhibited from being rewritten via the 
image sensing element 14, A/D converter 16. image 
processing circuit 20, and memory control circuit 22, 
and the latest written image data is displayed on the 



above-mentioned image display region of the image dis- 
play unit 28 via the memory control circuit 22 and D/A 
converter 26, thus displaying a frozen image on the 
electronic viewfinder. 

5 [0390] The system control circuit 50 executes distance 
measurement to adjust the focus of the photographing 
lens 310 on the object, and executes photometry to 
determine the aperture value and shutter speed (step 
S1 74). In photometry, the electronic flash is set if neces- 

10 sary. 

[0391 ] The details of the flow of this distance meas- 
urement/photometry process step Si 74 are the same 
as those in Fig. 5 that has already been described in the 
first embodiment, but will be explained again later using 

15 Fig. 5. 

[0392] Upon completion of the distance measure- 
ment/photometry process step S174, the system con- 
trol circuit 50 sets the display state of the image display 
unit 28 in a through display state (step S175), and the 
20 flow advances to step S176. Note that the through dis- 
play state in step S175 is the same as that in step S1 71 . 
[0393] If the shutter switch SW2 is not pressed (step 

5176) and the shutter switch SW1 is canceled (step 

5177) , the flow returns to step S134. 

25 [0394] If the shutter switch SW2 has been pressed 
(step S176). the system control circuit 50 sets the dis- 
play state of the image display unit 28 in a fixed -color 
display state (step S1 78). and the flow advances to step 
S179. 

30 [0395] In fixed-color display, in place of sensed image 
data, which has been written in the image display mem- 
ory 24 via the image sensing element 14, A/D converter 
16, image processing circuit 20, and memory control 
circuit 22, substitute fixed-color image data is displayed 

35 on the image display unit 28 via the memory control cir- 
cuit 22 and D/A converter 26. thus displaying a fixed- 
color image on the electronic viewfinder. 
[0396] The system control circuit 50 executes an 
image sensing process including an exposure process 

40 for writing sensed image data in the memory 30 via the 
image sensing element 14, A/D converter 16, image 
processing circuit 20, and memory control circuit 22 or 
directly from the A/D converter 16 via the memory con- 
trol circuit 22, and a development process for executing 

45 various processes by reading out image data written in 
the memory 30 using the memory control circuit 22 and 
image processing circuit 20 if required (step S1 79). 
[0397] The details of the flow of this image sensing 
process step S1 79 are the same as those in Fig. 6 that 

so has already been described in the first embodiment, but 
will be explained again later using Fig. 6. 
[0398] The system control circuit 50 executes a dis- 
play image process for reading out image data written in 
the memory 30 by the image sensing process in step 

55 S179. and transferring the readout data to the image 
display memory 24 via the memory control circuit 22 
(step S180). 

[0399] The details of the flow of this display image 
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process step S180 are the same as those in Fig. 7 that 
has already been described in the first embodiment, but 
will be explained again later using Fig. 7. 
[0400] The system control circuit 50 sets the display 
state of the image display unit 28 in a quick review dis- 
play state (step S181). and the flow advances to step 
S182. 

[0401] In quick review display, image data transferred 
to the image display memory 24 by the display image 
process in step S180 is displayed on the aforemen- 
tioned image sensing region of the image display unit 
28 via the memory control circuit 22 and D/A converter 
26. thus implementing an electronic viewfinder function 
that automatically plays back a sensed image. 
[0402] The system control circuit 50 executes a com- 
pression process in which various image processes are 
done using the memory control circuit 22 and image 
processing circuit 20 if required by reading out sensed 
image data written in the memory 30, or an image is 
compressed in correspondence with the selected mode 
using the compression/expansion circuit 32 (step 
Si 82). After that, the system control circuit 50 executes 
a recording process for writing image data in the record- 
ing medium 500 or 510 such as a memory card, com- 
pact flash card, or the like via the interface 90 or 94, and 
the connector 92 or 96 (step S183). 
[0403] The details of the flow of this compression 
process step S182 are the same as those in Fig. 8 that 
has already been described in the first embodiment, but 
will be explained again later using Fig. 8. 
[0404] When the image display unit 28 is ON, a mes- 
sage such as "BUSY" or the like that indicates that a 
write is in progress is displayed on the image display 
unit 28 during a write of image data on the recording 
medium 500 or 510. 

[0405] Furthermore, the indication unit 54 makes a 
recording medium write access indication by flickering 
an LED or the like in combination with the above mes- 
sage. 

[0406] The system control circuit 50 continues quick 
review display on the image display unit 28 until the 
shutter switch SW2 is released (step S184). With this 
display, the user can carefully check the sensed image 
while pressing the shutter switch SW2. 
[0407] If the shutter switch SW2 is released (step 
S184), the system control circuit 50 sets the panoramic 
image sensing start flag (step S1 85) to indicate that the 
first image has already been sensed in a series of pan- 
oramic 'rnage sensing processes. In this manner, as 
has beo- described in step S142, the image sensing 
direction can be inhibited from being reversed after the 
first image is sensed in the horizontal or vertical pano- 
ramic mode. 

[0408] The system control circuit 50 sets the variable 
n to be n = n + 1 to select the next image sensing region 
(step S186), and resets the variable m to m = 1 (step 
S187). Then, the flow returns to step S134 to repeat a 
series of panoramic image sensing processes. 



[0409] In this manner, after the control returns to step 
S134 upon releasing the shutter switch SW2. the next 
image sensing region is selected in step S146, the 
image sensing regions which include the previous 

5 image sensing region and display sensed images are 
set in the quick review display state, and the next image 
sensing region is set in the through display state, so as 
to display the sensed images and the next image to be 
sensed to overlap each other in step S148, thus allow- 

7 o ing the user to easily sense panoramic images. 

[0410] Figs. 14A and 14B that have already been 
described in the first embodiment show examples of the 
image sensing window before and after completion of 
sensing of the n-th image in the horizontal panoramic 

15 mode. 

[0411] Fig. 14A shows the image sensing window 
before sensing the first image, and Fig. 14B shows the 
image sensing window upon releasing the shutter 
switch SW2 after the first image is sensed. Although no 

20 dedicated explanatory view is available, before the shut- 
ter switch SW2 is released after the first image sensing, 
the first image sensing region in Fig. 14A on the image 
sensing window is set in a quick review display state. 
[0412] Figs. 15A and 15B that have already been 

25 described in the first embodiment show examples of the 
image sensing window before and after completion of 
sensing of the n-th image in the vertical panoramic 
mode. 

[0413] Fig. 15A shows the image sensing window 
30 before sensing the first image, and Fig. 15B shows the 
image sensing window upon releasing the shutter 
switch SW2 after the first image is sensed. Although no 
dedicated explanatory view is available, before the shut- 
ter switch SW2 is released after the f irst image sensing, 
35 the first image sensing region in Fig. 15A on the image 
sensing window is set in a quick review display state. 
[0414] Furthermore, Figs. 12A to 13B that have 
already been described in the first embodiment show 
examples of the image sensing window before and after 
40 completion of sensing of the n-th image in the 2 x 2 pan- 
oramic mode. 

[0415] Fig. 12A shows the image sensing window 
before the first image is sensed. Fig. 12B shows the 
image sensing window upon releasing the shutter 
45 switch SW2 after the first image is sensed, i.e., before 
the second image is sensed. 

[0416] Fig. 13A shows the image sensing window 
upon releasing. the shutter switch SW2 after the second 
image is sensed, i.e., before the third image is sensed. 

so and Fig. 13B shows the image sensing window upon 
canceling the shutter switch SW2 after the third image is 
sensed, i.e.. before the fourth image is sensed. 
[0417] Although no dedicated explanatory view is 
available, before the shutter switch SW2 is released 

55 after the n-th image sensing, the n-th image sensing 
region in Figs. 12A to 13B on the image sensing window 
is set in a quick review display state. When the shutter 
switch SW2 is released, a through display of the (n+1)- 
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th image sensing region as the next region is added. 
[0418] As mentioned above, according to the second 
embodiment, after a series of panoramic image sensing 
processes have started, zoom operation is inhibited, 
and a predetermined alert is generated, thus preventing 
an image unsuitable for panoramic synthesis from being 
sensed by zoom operation of the photographing lens 
310 during a series of panoramic image sensing proc- 
esses. 

[0419] Fig. 5 is a flow chart showing the flow of the 
operation of the distance measurement/photometry 
process in step Si 74 shown in Fig. 21 in detail. 
[0420] The system control circuit 50 reads out a 
charge signal from the image sensing element 14, and 
the image processing circuit 20 sequentially reads 
sensed image data via the A/D converter 16 (step 
S501). Using the read image data, the image process- 
ing circuit 20 makes predetermined computations used 
in the TTL (Through The Lens) AE (auto exposure) 
process, EF (flash pre-emission) process, and AF (auto 
focus) process. 

[0421 ] Note that the respective processes extract spe- 
cific portions at required positions from all pixels, and 
use them in computations. In this manner, in the TTL 
AE, EF, AWB, and AF processes, optimal computations 
can be made in units of different modes, i.e., a center- 
weighted mode, average mode, evaluation mode, and 
the like. 

[0422] The system control circuit 50 makes the expo- 
sure control means 340 execute AE control using the 
computation results of the image processing circuit 20 
(step S503) until it is determined that the exposure 
value (AE) is appropriate (step S502). 
[0423] The system control circuit 50 checks using 
measurement data obtained by the AE control if flash 
emission is required (step S504). If flash emission is 
required, the system control circuit 50 sets a flash flag to 
charge the electronic flash 48 (step S505). 
[0424] If it is determined that the exposure value (AE) 
is appropriate (step S502), the system control circuit 50 
stores the measurement data and/or setting parameters 
in. its internal memory or the memory 52. 
[0425] The system control circuit 50 makes the image 
processing circuit 20 execute AWB control by adjusting 
the color processing parameters (step S507) using the 
computation results of the image processing circuit 20 
and measurement data obtained by the AE control, until 
it is determined that the white balance (AWB) is appro- 
priate (step S506). 

[0426] If it is determined that the white balance (AWB) 
is appropriate (step S506), the system control circuit 50 
stores the measurement data and/or setting parameters 
in its internal memory or the memory 52. 
[0427] The system control circuit 50 makes the dis- 
tance measurement control means 342 execute AF 
control using measurement data obtained by the AE 
control and AWB control (step S509), until it is deter- 
mined that the distance measurement (AF) result indi- 



cates in-focus (step S508). 

[0428] If it is determined that the distance measure- 
ment (AF) result indicates in-focus (step S508), the sys- 
tem control circuit 50 stores the measurement data 

s and/or setting parameters in its internal memory or the 
memory 52, thus ending the distance measure- 
ment/photometry process routine step S174. 
[0429] Fig. 6 is a flow chart showing the image sens- 
ing process in step S179 in Fig. 21 in detail. 

w [0430] The system control circuit 50 opens the shutter 
312 having a stop function using the exposure control 
means 340 in accordance with photometry data stored 
in its internal memory or the memory 52 to expose the 
image sensing element 14 (steps S601 and S602). 

15 [0431 ] The system control circuit 50 checks based on 
the flash flag if the electronic flash 48 is required (step 
S603). If the electronic flash 48 is required, the system 
control circuit 50 controls the electronic flash to emit 
light (step S604). 

20 [0432] The system control circuit 50 waits for comple- 
tion of exposure of the image sensing element 14 in 
accordance with the photometry data (step S605), 
closes the shutter 312 (step S606), and reads a charge 
signal from the image sensing element 14 and writes 

25 sensed image data in the memory 30 via the A/D con- 
verter 16. image processing circuit 20, and memory 
control circuit 22 or directly from the A/D converter 16 
via the memory control circuit 22 (step S607). 
[0433] Upon completion of a series of processes, the 

30 image sensing process routine step S1 79 ends. 

[0434] Fig. 7 is a flow chart showing the display image 
process in step S180 in Fig. 21 in detail. 
[0435] If a frame process is required in correspond- 
ence with the selected image sensing mode (step 

35 S701), the system control circuit 50 reads out image 
data written in the memory 30, executes a vertical addi- 
tion process (step S702) and color process (step S703) 
in turn using the memory control circuit 22 and image 
processing apparatus 20 if required, and writes the 

40 processed image data in the memory 30. 

[0436] The system control circuit 50 reads out image 
data from the memory 30, and transfers display image 
data to the image display memory 24 via the memory 
control circuit 22 (step S704). 

45 [0437] Upon completion of a series of processes, the 
display image process routine step S180 ends. 
[0438] Fig. 8 is a flow chart showing the compression 
process in step S1 82 in Fig. 21 in detail. 
[0439] The^system control circuit 50 executes a pixel 

so squaring process for reading out sensed image data 
written in the memory 30. and converting the vertical-to- 
horizontal pixel ratio of the image sensing element to 1 
: 1 by interpolation using the memory control circuit 22 
and image processing circuit 20 if required (step S801). 

55 The system control circuit 50 then writes the processed 
image data in the memory 30. 

[0440] The system control circuit 50 reads out image 
data written in the memory 30, and executes image 
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compression corresponding to the selected mode using 
the compression/expansion circuit 32 (step S802). 
[0441] Upon completion of image compression, the 
compression process routine step S182 ends. 

(Third Embodiment) 

[0442] The operation of the third embodiment will be 
described below with reference to Figs. 22, 23, 24, 25, 
26, 5. 6, 7, and 8. The arrangement of the third embod- 
iment has the same appearance as that of the second 
embodiment, except for its operation. Also, the proc- 
esses shown in Figs. 5, 6, 7, and 8 are the same as 
those in the second embodiment, and a detailed 
description thereof will be omitted. 
-[0443] The operation of the third embodiment will be 
explained below using Figs. 22, 23, 24, 25, and 26. 
[0444] Upon power ON after battery exchange or the 
like, the system control circuit 50 initializes flags, control 
variables, and the like, and initializes the respective 
units of the image processing apparatus 400 (step 
S901). 

[0445] The system control circuit 50 checks the setting 
position of the mode dial 60. If the mode dial 60 is set at 
a power OFF position (step S902), the system control 
circuit 50 executes a predetermined end process (step 
S903). More specifically, the system control circuit 50 
changes the indications of the respective indication 
units to an end state, protects an image sensing unit by 
closing a barrier if the lens unit 300 comprises the bar- 
rier or the like serving as a protection means, records 
required parameters and setting values including flags, 
control variables, and the like, and the setting mode in 
the nonvolatile memory 56, cuts off unnecessary power 
supply to the respective units of the image processing 
apparatus 400 including the image display unit 28 by the 
power supply control means 80, and so forth. After that 
the flow returns to step S902. 

[0446] If the mode dial 60 is set at any of other mode 
positions (step S902), the system control circuit 50 
checks using the power supply control means 80 if the 
remaining capacity and operation state of the power 
supply means 86 comprising batteries and the like pose 
any problem in the operation of the image processing 
apparatus 400 (step S904). If any problem is found, a 
predetermined alert indication is made by means of an 
image or voice using the indication unit 54 (step S906), 
and the flow then returns to step S902. 
[0447] When the image display of the image display 
unit 28 is ON, a predetermined alert display is made by 
means of an image or voice also using the image dis- 
play unit 28. 

[0448] If no problem is found in the power supply 
means 86 (step S904). the system control circuit 50 
checks if the operation state of the recording medium 
500 or 510 poses any problem in the operation of the 
image processing apparatus 400, in particular, record- 
ing/playback of image data to/from the recording 



medium (step S905). rf any problem is found, a prede- 
termined alert indication is made by means of an image 
or voice using the indication unit 54 (step S906), and the 
flow then returns to step S902. 
5 [0449] When the image display of the image display 
unit 28 is ON, a predetermined alert display is made by 
means of an image or voice also using the image dis- 
play unit 28. 

[0450] If no problem is found in the operation state of 
10 the recording medium 500 or 510 (step S905), various 
setting states of the image display apparatus 400 are 
indicated by means of an image or voice using the indi- 
cation unit 54 (step S907). When the image display of 
the image display unit 28 is ON, various setting states of 
75 the image display apparatus 400 are indicated by 
means of an image or voice also using the image dis- 
play unit 28. 

[0451 ] On the other hand, if the mode dial 60 is set at 
an image sensing mode position (step S908), a prede- 
20 termined image sensing process is executed (step 
S909). Upon completion of the process, the flow returns 
to step S902. 

[0452] If the mode dial 60 is set at any of other mode 
positions (step S908), the system control circuit 50 exe- 
25 cutes a process corresponding to the selected mode 
(step $910). Upon completion of the process, the flow 
returns to step S902. 

[0453] If the mode dial 60 is set at a panoramic mode 
position (step S908), the system control circuit 50 

30 checks if the zoom position of the photographing lens 
310 is a predetermined home position (step S921). If 
the photographing lens is not located at the home posi- 
tion, the system control circuit 50 moves the zoom posi- 
tion to the home position (step S922). 

35 [0454] Also, when the photographing lens 310 uses a 
selectable lens that can select one of a plurality of differ- 
ent focal lengths, the system control circuit 50 checks if 
that selectable lens of the photographing lens 310 is 
located on the predetermined home position side (step 

40 S921). rf the selectable lens is not located on the home 
position side, the system control circuit 50 moves the 
selectable lens to the home position side (step S922). 
[0455] As an example of the home position, in case of 
a zoom lens, the WIDE end as a focal length position on 

45 the wide-angle side suitable for panoramic image sens- 
ing can be used. 

[0456] In case of a focal length selectable lens, a (ens 
with a focal length on the WIDE side as that on the wide- 
angle end suitable for panoramic image sensing may be 
so selected. 

[0457] The present invention is not limited to such 
specific example, and any focal length may be set at a 
home position as long as it is suitable for panoramic 
image sensing. 
55 [0458] The system control circuit 50 displays a pano- 
ramic mode selection window on the image display unit 
(step S911). 

[0459] The user of the image processing apparatus 
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400 can select an arbitrary one of a plurality of pano- 
ramic image sensing modes by observing the pano- 
ramic mode selection window displayed on the image 
display unit 28. 

[0460] The panoramic image sensing modes include 5 
a 2 x 2 panoramic mode, horizontal panoramic mode, 
vertical panoramic mode, and the like. In these modes, 
by combining a plurality of sensed images, a panoramic 
image, which is extended two-dimensionally, horizon- 
tally, or vertically, can be obtained. w 
[0461 ] A series of sensed images are appended with 
information indicating their positional relationship, and 
can be combined to generate a signal panoramic image 
by the image processing apparatus 400 itself or after 
they are transferred to, e.g., a computer or the like. is 
[0462] Note that an example of the panoramic mode 
selection window is the same as that shown in Fig. 9 
that has already been explained. 
[0463] Referring to Fig. 9, the left icon is used for 
selecting the 2 x 2 panoramic mode for sensing a two- 20 
dimensional panoramic image, the central icon is used 
for selecting the horizontal panoramic mode for sensing 
a horizontal panoramic image, and the right icon is used 
for selecting a vertical panoramic mode for sensing a 
vertical panoramic image. 2s 
[0464] The user of the image processing apparatus 
400 can select a desired panoramic mode by the selec- 
tion/change switch 66 (e.g.. [+] key), and can determine 
that panoramic mode by the determination/execution 
switch 68 (e.g., [SET] key). 30 
[0465] When the panoramic mode is selected using 
the selection/change switch 66 and determination/exe- 
cution switch 68, the system control circuit 50 sets initial 
values and memory areas of flags, parameters, and var- 
iables, reads out a display window serving as a user 35 
interface, and so forth in a 2 x 2 panoramic mode initial 
setup process (step S913), a vertical panoramic mode 
initial setup process (step S914), or a horizontal pano- 
ramic mode initial setup process (step S915) in corre- 
spondence with the selected panoramic mode (step 40 
S912), and displays a panoramic image sensing win- 
dow corresponding to the selected mode on the image 
display unit 28 (step S916). The flow then advances to 
step S931. 

[0466] Note that Figs. 1 0A and 1 0B that have already 45 
been explained show examples of the horizontal pano- 
ramic mode image sensing window, Figs. 1 1 A and 1 1 B 
show examples of the vertical panoramic mode image 
sensing window, and Figs. 12A and 12B that have 
already been explained show examples of the 2 x 2 so 
panoramic mode image sensing window. 
[0467] Fig. 10A shows an image sensing window that 
creates a panoramic window by continuously repeating 
image sensing to the right, and Fig. 10B shows an 
image sensing window that creates a panoramic win- 55 
dow by continuously repeating image sensing to the left. 
[0468] Referring to Figs. 10A and 10B, a through 
image from the electronic viewfinder is displayed on 



region "1" to sense the first image of a series of pano- 
ramic images. Before completion of image sensing of 
the first image, rightward image sensing (Fig. 10A) or 
leftward image sensing (Fig. 10B) can be desirably 
selected using the selection/change switch 66 (e.g., [+] 
key). 

[0469] The image sensing direction is determined by 
sensing the first image by pressing the shutter button in 
desired one of the display window states shown in Figs. 
1 0A and 1 0B. 

[0470] Note that the user can end image sensing in 
the horizontal panoramic mode by pressing the determi- 
nation/execution switch 68 (e.g., [SET] key). 
[0471 ] Fig. 1 1 A shows an image sensing window that 
creates a panoramic window by continuously repeating 
image sensing upward, and Fig. 11B shows an image 
sensing window that creates a panoramic window by 
continuously repeating image sensing downward. 
[0472] Referring to Figs. 11A and 11B, a through 
image from the electronic viewfinder is displayed on 
region "1" to sense the first image of a series of pano- 
ramic images. Before completion of image sensing of 
the first image, upward image sensing (Fig. 11 A) or 
downward image sensing (Fig. 11B) can be desirably 
selected using the selection/change switch 66 (e.g.. [+] 
key). 

[0473] The image sensing direction is determined by 
sensing the first image by pressing the shutter button in 
desired one of the display window states shown in Figs. 
llAand 11B. 

[0474] Note that the user can end image sensing in 
the vertical panoramic mode by pressing the determina- 
tion/execution switch 68 (e.g.. [SET] key). 
[0475] Fig. 12A shows an image sensing window for 
the first image upon creating a 2 x 2 panoramic window 
by two-dimensionally repeating image sensing of a total 
of four images in the vertical and horizontal directions, 
and Fig. 12B shows an image sensing window for the 
second image. 

[0476] Referring to Figs. 12A and 12B, a through 
image from the electronic viewfinder is displayed on, 
e.g.. an upper right region to sense the first one of pan- 
oramic images. 

[0477] Upon completion of image sensing of the first 
image, the playback image of the sensed image is dis- 
played on, e.g., the upper right region, and a through 
image from the electronic viewfinder is displayed on, 
e.g., the lower right region to sense the second image. 
[0478] njhe image sensing region of the electronic 
viewfinder is switched in turn to the next region after 
each image sensing. However, image sensing may be 
started from an arbitrary region, or may be redone by 
returning to the already sensed region. For this pur- 
pose, the image sensing region can be desirably shifted 
using the selection/change switch 66 (e.g., [+] switch). 
[0479] The image sensing region to be re-sensed is 
determined and image sensing is executed by pressing 
the shutter button in a display window state that selects 
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a desired region. 

[0480] Note that the user can end image sensing in 
the 2 x 2 panoramic mode by pressing the determina- 
tion/execution switch 68 (e.g., [SET] key). 
[0481] The system control circuit 50 initializes flags 
and variables set in its internal memory or the memory 
52. For example, the system control circuit 50 resets a 
panoramic image sensing start flag (step S931), sets 
variables m and n, and initializes them to m = 1 and n = 
1 , respectively (steps S933 and S932). 
[0482] Note that the panoramic image sensing start 
flag is reset to indicate that no panoramic image sens- 
ing processes have been done. 
[0483] Also, the variable n indicates the position of the 
current image sensing region in image sensing of each 
panoramic mode. 

[0484] The variable m is used for returning the image 
sensing region of interest to the immediately preceding 
region or canceling such return, upon executing image 
sensing again in the horizontal and vertical panoramic 
modes. 

[0485] An example of the method of using these flag 
and variables will be explained later. 
[0486] rf the mode dial 60 is set at any of mode posi- 
tions other than the panoramic mode (step S934), the 
system control circuit 50 ends image sensing in the pan- 
oramic mode, and the flow returns to step S902. In this 
manner, the user of the image processing apparatus 
400 can end the panoramic mode anytime he or she 
wants, and can select another mode. 
[0487] If end of the panoramic mode is selected by 
pressing the determination/execution switch 68 (e.g., 
[SET] key) (step S935) while the mode dial 60 is kept 
set at the panoramic mode position (step S934), i.e., the 
display window of the image display unit 28 keeps dis- 
playing the corresponding panoramic image sensing 
window, a panoramic mode end confirmation message 
is displayed. Furthermore, if confirmation of end is 
selected (step S936), the flow returns to step S91 1 of 
displaying the panoramic mode selection window. 
[0488] If the user has pressed the zoom switch WIDE 
74 or zoom switch TELE 76 (step S961), the system 
control circuit 50 checks the status of the panoramic 
image sensing start flag (step S962). 
[0489] If the panoramic image sensing start flag is 
reset, i.e., if the first image is not sensed yet in a series 
of panoramic image sensing processes (step S962), the 
system control circuit 50 executes predetermined zoom 
operation for zooming the photographing lens 310 via 
the interlace 420. connector 422. connector 322. inter- 
face 320. lens system control circuit 350, and zoom 
control means 344 in accordance with an instruction 
input by the zoom switch WIDE 74 or zoom switch TELE 
76 (step S963). The flow then advances to step S937. 
[0490] If the panoramic image sensing start flag is set. 
i.e., rf the first image has already been sensed in a 
series of panoramic image sensing processes (step 
S962). the system control circuit 50 makes a predeter- 



mined alert display by means of an image or voice using 
the indication unit 54 and/or the image display unit 28 
(step S964). Furthermore, if confirmation of the end of a 
series of panoramic image sensing processes is 
5 selected (step S965), the system control circuit 50 tem- 
porarily ends the series of panoramic image sensing 
processes executed so far, and the flow returns to step 
S931 to start a new series of panoramic image sensing 
processes. 

io [0491] On the other hand, if cancel of the end of a 
series of panoramic image sensing processes is 
selected (step S965), the flow advances to step S937 
without executing any zoom operation. 
[0492] With this control, an image which is unsuitable 

is for panoramic synthesis can be prevented from being 
sensed by zoom operation of the photographing lens 
310 during a series of panoramic image sensing proc- 
esses. 

[0493] If the selection/change switch 66 (e.g., [+] key) 
20 has been pressed while the image display unit 28 dis- 
plays one of the above-mentioned panoramic image 
sensing windows (step S937), the system control circuit 
50 executes a predetermined process in correspond- 
ence with the setups of the panoramic mode (step 
25 S938). 

- [0494] If the selection/change switch 66 (e.g.. [+] key) 
is not pressed (step S937), the flow advances to step 
S946. 

[0495] If the 2 x 2 panoramic mode is selected (step 
30 S938), the system control circuit 50 sets the variable n 
to be n = n + 1 to select the next image sensing region 
(step S939). If n = 5 (step S940), the first image sensing 
region is selected by setting n = n - 4 (step S941). After 
that, the flow advances to step S946. 
35 [0496] More specifically, every time the selec- 
tion/change switch 66 (e.g., [+] key) is pressed, the 
image sensing region of interest shifts to one of the four 
image sensing regions in the 2 x 2 panoramic mode. For 
example, the image sensing region of interest shifts in 
40 the order of the upper right region, lower right region, 
lower left region, upper left region, upper right region, 
lower right region 

[0497] On the other hand, if the horizontal or vertical 
panoramic mode is selected (step S938). and if the pan- 

45 oramic image sensing start flag is reset, i.e., if the first 
image is not sensed yet in a series of panoramic image 
sensing processes (step S942), the system control cir- 
cuit 50 reverses the image sensing direction of the 
selected panoramic mode (step S943). After that, the 

so flow advances to step S946. 

[0498] Note that the image sensing direction can be 
repetitively reversed before the first image is sensed. 
[0499] Figs. 10A and 10B that have already been 
described show examples of an image sensing direction 

55 reverse window in the horizontal panoramic mode. 

[0500] Fig. 10A shows a window upon repeating 
image sensing to the right, and Fig. 10B shows a win- 
dow upon repeating image sensing to the left. 
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[0501] Figs. 11A and 11B that have already been 
described show examples of an image sensing direction 
reverse window in the vertical panoramic mode. 
10502] Fig. 11A shows a window upon repeating 
tmage sensing upward, and Fig. 11B shows a window 5 
upon repeating image sensing downward. 
[0503] On the other hand, if the horizontal or vertical 
panoramic mode is selected (step S938) and if the pan- 
oramic image sensing start flag is set, i.e., if the first 
image has already been sensed in a series of pano- w 
ramie image sensing processes (step S942), the sys- 
tem control circuit 50 computes m = m x (-1), the sign of 
which changes like -1, +1, -1, +1,... in every computa- 
tion (step S944), and also computes n = n + m for the 
variable n, which changes like n = n-1,n = n+1,n = n 15 
-T.n»n+1,... in every computation (step S945). After 
that, the flow advances to step S946. 
[0504] With this control, movement to the already 
sensed region (which is limited to the one sensed last) 
or cancel that movement can be repetitively selected to 20 
sense an image again. 

[0505] After panoramic image sensing of the n-th 
image region, the user presses the selection/change 
switch 65 (e.g.. [+] key) to confirm if quick review images 
of the (n-l)-th and n-th image regions smoothly join at 25 
their boundary regions. Then, the user temporarily 
releases the selection/change switch 66 (e.g., [+] key) to 
display a through image on the n-th image region, and 
presses the selection/change switch 66 (e.g., [+] key) 
once again to display the original quick review image on 30 
the n-th image region and a through image on the (n+1 )- 
th image region, thus executing the next panoramic 
image sensing. 

[0506] After panoramic image sensing of the n-th 
image region, upon pressing the selection/change 35 
switch 66 (e.g., [+] key) to confirm if quick review images 
of the (n-1)-th and n-th image regions smoothly join at 
their boundary regions, if the user determines that 
image sensing must be redone, he or she temporarily 
releases the selection/change switch 66 (e.g. , [+] key) to 40 
display a through image on the n-th image region, and 
presses the shutter switch SW2 in that window state to 
replace the image with the newly sensed image. Then, 
the user can execute the next panoramic image sens- 
ing. 45 
[0507] The system control circuit 50 selects the n-th 
image region (step S946). If the window layout must be 
changed in correspondence with the selected image 
region (step S947), the system control circuit 50 
switches and displays a new window layout (step S948) so 
The flow then advances to step S971 . 
[0508] In this fashion, a window corresponding to the 
reversed image sensing direction, as described above, 
a window corresponding to the changed image sensing 
region, a window that has updated characters and 55 
images for comments, or the like can be displayed as 
needed. 

[0509] Upon sensing the second and subsequent 



images, the image sensing regions that display already 
sensed images are set in a quick review display state, 
and the image sensing region selected for the next 
image sensing is set in a through display state, thus dis- 
playing a playback image of the already sensed image 
and the next image to be sensed to partially overlap 
each other. In this manner, the user of the image 
processing apparatus 400 can easily determine the next 
image sensing angle. 

[0510] The system control circuit 50 sets a through 
display state in which sensed image data is displayed 
as it is sensed (step S971), and the flow advances to 
step S972. 

[051 1 ] In through display, data, which have been writ- 
ten in the image display memory 24 via the image sens- 
ing element 14, A/D converter 16, image processing 
circuit 20, and memory control circuit 22, are displayed 
on the aforementioned image sensing region of the 
image display unit 28 via the memory control circuit 22 
and D/A converter 26, thereby implementing an elec- 
tronic viewfinder function. 

[0512] rf the shutter switch SW1 is not pressed (step 
S972), the flow returns to step S934. 
[0513] If the shutter switch SW1 has been pressed 
(step S972). the system control circuit 50 sets the dis- 
play state of the image display unit 28 in a freeze display 
state (step S973). and the flow advances to step S974. 
[0514] In freeze display, image data on the image dis- 
play memory 24 is inhibited from being rewritten via the 
image sensing element 14. A/D converter 16, image 
processing circuit 20. and memory control circuit 22, 
and the latest written image data is displayed on the 
above-mentioned image display region of the image dis- 
play unit 28 via the memory control circuit 22 and D/A 
converter 26. thus displaying a frozen image on the 
electronic viewfinder. 

[051 5] The system control circuit 50 executes distance 
measurement to adjust the focus of the photographing 
lens 310 on the object, and executes photometry to 
determine the aperture value and shutter speed (step 
S974). In photometry, the electronic flash is set if neces- 
sary. 

[0516] The details of the flow of this distance meas- 
urement/photometry process step S974 are the same 
as those in the second embodiment that has already 
been described using Fig. 5. 

[0517] Upon completion of the distance measure- 
ment/photometry process step S974, the system con- 
trol circuit 59 sets the display state of the image display 
unit 28 in a through display state (step S975), and the 
flow advances to step S976. Note that the through dis- 
play state in step S975 is the same as that in step S971 . 
[0518] If the shutter switch SW2 is not pressed (step 

5976) and the shutter switch SW1 is canceled (step 

5977) . the flow returns to step S934. 

[0519] ff the shutter switch SW2 has been pressed 
(step S976), the system control circuit 50 sets the dis- 
play state of the image display unit 28 in a fixed-color 
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display state (step S978) r and the flow advances to step 
S979. 

[0520] In fixed-color display, in place of sensed image 
data, which has been written in the image display mem- 
ory 24 via the image sensing element 14, A/D converter 
16, image processing circuit 20, and memory control 
circuit 22, substitute fixed-color image data is displayed 
on the image display unit 28 via the memory control cir- 
cuit 22 and D/A converter 26, thus displaying a fixed- 
color image on the electronic viewfinder. 
[0521] The system control circuit 50 executes an 
image sensing process including an exposure process 
for writing sensed image data in the memory 30 via the 
image sensing element 14, A/D converter 16, image 
processing circuit 20, and memory control circuit 22 or 
directly from the A/D converter 16 via the memory con- 
trol circuit 22, and a development process for executing 
various processes by reading out image data written in 
the memory 30 using the memory control circuit 22 and 
image processing circuit 20 if required (step S979). 
[0522] The details of the flow of this image sensing 
process step S979 are the same as those in the second 
embodiment that has already been described using Fig. 
6. 

[0523] The system control circuit 50 executes a dis- 
play image process for reading out image data written in 
the memory 30 by the image sensing process in step 
S979, and transferring the readout data to the image 
display memory 24 via the memory control circuit 22 
(step S980). 

[0524] The details of the flow of this display image 
process step S980 are the same as those in the second 
embodiment that has already been described using Fig. 
7. 

[0525] The system control circuit 50 sets the display 
state of the image display unit 28 in a quick review dis- 
play state (step S981), and the flow advances to step 
S982. 

[0526] In quick review display, image data transferred 
to the image display memory 24 by the display image 
process in step S980 is displayed on the aforemen- 
tioned image sensing region of the image display unit 
28 via the memory control circuit 22 and D/A converter 
26, thus implementing an electronic viewfinder function 
that automatically plays back a sensed image. 
[0527] The system control circuit 50 executes a com- 
pression process in which various image processes are 
done using the memory control circuit 22 and image 
processing circuit 20 if required by reading out sensed 
image data written in the memory 30, or an image is 
compressed in correspondence with the selected mode 
using the compression/expansion circuit 32 (step 
S982). After that, the system control circuit 50 executes 
a recording process for writing image data in the record- 
ing medium 500 or 510 such as a memory card, com- 
pact flash card, or the like via the interface 90 or 94, and 
the connector 92 or 96 (step S983). 
[0528] The details of the flow of this compression 



process step S982 are the same as those in the second 
embodiment that has already been described using Fig. 
8. 

[0529] When the image display unit 28 is ON. a mes- 
5 sage such as "BUSY" or the like that indicates that a 
write is in progress is displayed on the image display 
unit 28 during a write of image data on the recording 
medium 500 or 510. 

[0530] Furthermore, the indication unit 54 makes a 
10 recording medium write access indication by flickering 
an LED or the like in combination with the above mes- 
sage. 

[0531] The system control circuit 50 continues quick 
review display on the image display unit 28 until the 

is shutter switch SW2 is released (step S984). With this 
display, the user can carefully check the sensed image 
while pressing the shutter switch SW2. 
[0532] If the shutter switch SW2 is released (step 
S984), the system control circuit 50 sets the panoramic 

20 image sensing start flag (step S985) to indicate that the 
first image has already been sensed in a series of pan- 
oramic image sensing processes. In this manner, as 
has been described in step S942, the image sensing 
direction can be inhibited from being reversed after the 

25 first image is sensed in the horizontal or vertical pano- 
ramic mode. 

[0533] The system control circuit 50 sets the variable 
n to be n = n + 1 to select the next image sensing region 
(step S986). and resets the variable m to m = 1 (step 

30 S987). Then, the flow returns to step S934 to repeat a 
series of panoramic image sensing processes. 
[0534] In this manner, after the control returns to step 
S934 upon releasing the shutter switch SW2, the next 
image sensing region is selected in step S946, the 

35 image sensing regions which include the previous 
image sensing region and display sensed images are 
set in the quick review display state, and the next image 
sensing region is set in the through display state, so as 
to display the sensed images and the next image to be 

40 sensed to overlap each other in step S948, thus allow- 
ing the user to easily sense panoramic images. 
[0535] Figs. 14A and 14B that have already been 
described show examples of the image sensing window 
before and after completion of sensing of the n-th image 

45 in the horizontal panoramic mode. 

[0536] Fig. 14A shows the image sensing window 
before sensing the first image, and Fig. 14B shows the 
image sensing, window upon releasing the shutter 
switch SW2 after the first image is sensed. Although no 

so dedicated explanatory view is available, before the shut- 
ter switch SW2 is released after the first image sensing, 
the first image sensing region in Fig. 14A on the image 
sensing window is set in a quick review display state. 
[0537] Figs. 15A and 15B that have already been 

55 described show examples of the image sensing window 
before and after completion of sensing of the n-th image 
in the vertical panoramic mode. 
[0538] Fig. 15A shows the image sensing window 
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before sensing the first image, and Fig. 15B shows the 
image sensing window upon releasing the shutter 
switch SW2 after the first image is sensed. Although no 
dedicated explanatory view is available, before the shut- 
ter switch SW2 is released after the first image sensing, 
the first image sensing region in Fig. 15A on the image 
sensing window is set in a quick review display state. 
[0539] Furthermore, Figs. 12A to 13B that have 
already been described show examples of the image 
sensing window before and after completion of sensing 
of the n-th image in the 2 x 2 panoramic mode. 
[0540] Fig. 12A shows the image sensing window 
before the first image is sensed. Fig. 12B shows the 
image sensing window upon releasing the shutter 
switch SW2 after the first image is sensed, i.e., before 
the second image is sensed. 

[0541] Fig. 13A shows the image sensing window 
upon releasing the shutter switch SW2 after the second 
image is sensed, i.e., before the third image is sensed, 
and Fig. 13B shows the image sensing window upon 
canceling the shutter switch SW2 after the third image is 
sensed, i.e., before the fourth image is sensed. 
[0542] Although no dedicated explanatory view is 
available, before the shutter switch SW2 is released 
after the n-th image sensing, the n-th image sensing 
region in Figs. 12A to 13B on the image sensing window 
is set in a quick review display state. When the shutter 
switch SW2 is released, a through display of the (n+1)- 
th image sensing region as the next region is added. 
[0543] As mentioned above, according to the third 
embodiment, when 200m operation is done after the 
start of a series of panoramic image sensing processes, 
a predetermined alert is generated to ask the user of the 
image processing apparatus 400 if he or. she actually 
wants to zoom. After that, a series of panoramic image 
sensing processes are temporarily interrupted. Thus, an 
image which is unsuitable for panoramic synthesis can 
be prevented from being sensed by zoom operation of 
the photographing lens 310 during a series of pano- 
ramic image sensing processes. 
[0544] The second and third embodiments have been 
explained using Fig. 16. 

[0545] In the description of the second and third 
embodiments, the photographing lens 310 uses a zoom 
lens, but may use a selectable lens that can select one 
of a wide-angle lens and telephoto lens. In this case, as 
in the description of the embodiments, execution of lens 
selection is inhibited in correspondence with the state of 
the panoramic image sensing start flag, and a predeter- 
mined alert is generated. Or after a predetermined alert 
is generated to ask the user if he or she actually wants 
such lens selection, a series of panoramic image sens- 
ing processes can be ended, and a new series of pano- 
ramic image sensing processes can be started. 
[0546] In the description of the third embodiment, if 
the end of a series of panoramic image sensing proc- 
esses is confirmed in step S965, the flow returns to step 
S931 to start a new series of panoramic image sensing 



processes. Alternatively, the flow may return to step 
S921 to restart setups from the panoramic mode selec- 
tion window, and then a new series of panoramic image 
sensing processes may be started. 

5 [0547] Note that the number of types of selectable 
panoramic modes on the panoramic mode selection 
window is 3, i.e.. the 2 x 2 panoramic mode, horizontal 
panoramic mode, and vertical panoramic mode. How- 
ever, the present invention is not limited to these spe- 

10 crfic modes, and one or a plurality of types of an 
arbitrary number of different panoramic modes may be 
selectable. 

[0548] Also, display of the panoramic mode selection 
window shown in Fig. 9 is not limited to this example. 
15 Alternatively, display with an arbitrary layout or scheme 
using an image or voice may be made. 
[0549] On the 2 x 2 panoramic mode image sensing 
window, a total of four images are two-dimensionally 
laid out in a 2 x 2 matrix. However, the present invention 
20 is not limited to such matrix. For example, an arbitrary 
number of images such as a total of nine images, a total 
of 16 images, and the like may be laid out in a two- 
dimensional or three-dimensional matrix. 
[0550] On the horizontal and vertical panoramic mode 
25 image sensing windows, two images are continuously 
displayed. However, the present invention is not limited 
to such specific display, and an arbitrary number of 
images (e.g.. three images, four images, and the like) 
may be one-dimensionaily displayed. 
30 [0551 ] The displays of the panoramic mode image 
sensing windows shown in Figs. 10A to 15B are not lim- 
ited to these examples, and displays with arbitrary lay- 
outs and schemes using images, voices, characters, 
symbols, and the like may be made. 
35 [0552] In the horizontal and vertical panoramic 
modes, the image sensing direction is inhibited from 
being reversed after the first image is sensed. However, 
the present invention is not limited to this, and the image 
sensing direction may be reversed as needed. Also, the 
40 image sensing direction may be changed in one of arbi- 
trary directions including the right and left directions, up 
and down directions, or oblique directions. In the 
present invention, when a large number of image dis- 
play regions can be simultaneously displayed by 
45 increasing the display area of the image display unit 28, 
the image sensing direction can be changed in one of 
arbitrary directions including the right and left directions, 
up and down directions, or oblique directions. 
[0553] Image sensing can be redone for only the 
so immediately preceding sensed image in the horizontal 
and vertical panoramic modes. However, the present 
invention is not limited to this. For example, image sens- 
ing may be redone for an image sensed an arbitrary 
number of images before. 
55 [0554] The next image display region for displaying an 
image to be sensed is set in the through display state, 
and that for displaying the already sensed image is set 
in the quick review display state, to display a playback 
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image of the already sensed image and the next image 
to be sensed so that their boundary regions partially 
overlap each other. However, the present invention is 
not limited to such specific scheme. For example, the 
images may be displayed without making their bound- 
ary regions overlap each other. 
(0555] In the explanatory views of the panoramic 
modes in Figs. 10A to 15B, for example, comment 
words "LIVE" and "FREEZE" are added within the 
image display regions. However, the actual image dis- 
play unit 28 may or may not display these comment 
words. 

{0556] In the above description, the lens unit 300 is 
independent from the image processing apparatus 400 
and rs arbitrarily connectable. However, the lens unit 
300 may be integrated with the image processing appa- 
ratus 400 

[0557] Furthermore, a plurality of lens units 300 may 
be detachaWy attached to the image processing appa- 
ratus 400 

[0558] The recording media 500 and 510 are not lim- 
ited to memory cards such as PCMCIA cards, compact 
flash cards, or the like, hard disks, and the like, but may 
use rmcro DATs (digital audio tapes), magnetooptical 
disks, optical disks such as CD-Rs, CD-WRs. or the like, 
phase change optical disks such as DVDs, and the like. 
[0559] Also, the recording media 500 and 510 may 
use hybrid media that integrate memory cards, hard 
disks, and the like. Furthermore, such hybrid media may 
include detachable media. 

[0560] In the description of the second and third 
embodiments, the recording media 500 and 510 are 
independent from the image processing apparatus 400 
and are arbitrarily connectable. One or both the record- 
ing media 500 and 510 may be permanently connected 
to the image processing apparatus 400. When all the 
recording media are permanently connected to the 
image processing apparatus 400, no alert display is 
0%?;? xmade -in-correspondence withthe presence/absence of 
^s^^recortingr media. However, when all the recording 
media are incapable of recording due to failures, a 
recording inhibition setup state, no free recording area, 
and the like, the same effect as in the above embodi- 
ments can be obtained. 

[0561] An arbitrary number (one or a plurality) of 
image recording media 500 or 510 may be connectable 
to the image processing apparatus 400. 
[0562] In the above description, the recording media 
500 and 510 are attached to the image processing 
apparatus. However, one or a combination of a plurality 
of recording media may be used. 
[0563] In the description of the second and third 
embodiments, the focal length is changed as a change- 
able condition of the optical system of the photograph- 
ing lens. However, the present invention is not limited to 
such specific condition. For example, the changeable 
condition of the optical system includes exchange of 
lenses, stop, focus position, shutter speed, and the like, 



and the present invention can be applied to such 
changes in condition of the optical system. 
[0564] Synthesis of a panoramic image has been 
described as an example in which a plurality of images 

5 are associated. Also, the present invention can be 
applied to a case wherein a plurality of images are 
sensed while shifting pixels, and a high-resolution 
image is obtained by synthesizing these images. 
[0565] To restate, the second and third embodiments 

10 can solve the conventional problem that the image qual- 
ity deteriorates considerably due to extra correction 
computations required to correct different focal lengths 
upon generating a panoramic image by combining a 
plurality of sensed images when the user of the image 

is processing apparatus such as an electronic camera or 
the like switches the focal length of the photographing 
lens or performs zooming. 

[0566] Also, the second and third embodiments can 
solve the conventional problems that the outer shape of 
20 a synthesized panoramic image varies in units of 
sensed image and has steps when the individual 
sensed images have considerably different angles due 
to switching of the focal length of the photographing lens 
or zooming. 

25 [0567] Furthermore, the second and third embodi- 
ments can remedy a situation in the conventional art 
where the aforementioned problems readily occur when 
the user switches the focal length of the photographing 
lens or zooms even during sensing of a plurality of 

30 images for obtaining a panoramic image if he or she 
notices during panoramic image sensing that the focal 
length of the photographing lens is switched to a side 
unsuitable for panoramic image sensing or that the 
zoom position of the zoom lens is unsuitable for pano- 

35 ramie image sensing. 

(Fourth Embodiment) 

[0568] The operation of the fourth embodiment will be 

40 described below with reference to Figs. 27. 28, 29. 30, 
5, e' 7. and 8. The arrangement of the fourth embodi- 
ment has the same appearance as that of the second 
embodiment, except for its operation. Also, the proc- 
esses shown in Figs. 5, 6, 7, and 8 are the same as 

45 those in the second embodiment, and a detailed 
description thereof will be omitted. 
[0569] Figs. 27. 28, 29, and 30 are flow charts show- 
ing the main routine of the image processing apparatus 
400 of this embodiment. 

so [0570] The operation of the third embodiment will be 
explained below using Figs. 27, 28, 29, and 30. 
[0571] Upon power ON after battery exchange or the 
like, the system control circuit 50 initializes flags, control 
variables, and the like, and initializes the respective 

55 units of the image processing apparatus 400 (step 
S1101). 

[0572] The system control circuit 50 checks the setting 
position of the mode dial 60. If the mode dial 60 is set at 
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a power OFF position (step S1 102), the system control 
circuit 50 executes a predetermined end process (step 
S1 103). More specifically, the system control circuit 50 
changes the indications of the respective indication 
units to an end state, protects an image sensing unit by 
closing a barrier if the lens unit 300 comprises the bar- 
rier or the like serving as a protection means, records 
required parameters and setting values including flags, 
control variables, and the like, and the setting mode in 
the nonvolatile memory 56. cuts off unnecessary power 
supply to the respective units of the image processing 
apparatus 400 including the image display unit 28 by the 
power supply control means 80, and so forth. After that, 
the flow returns to step S1 102. 

[05731 If the mode dial 60 is set at any of other mode 
positions (step S1 102), the system control circuit 50 
checks using the power supply control means 80 if the 
remaining capacity and operation state of the power 
supply means 86 comprising batteries and the like pose 
any problem in the operation of the image processing 
apparatus 400 (step S1104). If any problem is found, a 
predetermined alert indication is made by means of an 
image or voice using the indication unit 54 (step S1 1 06), 
and the flow then returns to step S1102. 
[0574] When the image display of the image display 
unit 28 is ON. a predetermined alert display is made by 
means of an image or voice also using the image dis- 
play unit 28. 

[0575] If no problem is found in the power supply 
means 86 (step S1104), the system control circuit 50 
checks if the operation state of the recording medium 
500 or 510 poses any problem in the operation of the 
image processing apparatus 400, in particular, record- 
ing/playback of image data to/from the recording 
medium (step S1 105). If any problem is found, a prede- 
termined alert indication is made by means of an image 
or voice using the indication unit 54 (step S1106), and 
the flow then returns to step S1 102. 
[0576] When the image display of the image display 
unit 28 is ON, a predetermined alert display is made by 
means of an image or voice also using the image dis- 
play unit 28. 

[0577] If no problem is found in the operation state of 
the recording medium 500 or 510 (step S1 105), various 
setting states of the image display apparatus 400 are 
indicated by means of an image or voice using the indi- 
cation unit 54 (step S1 107). When the image display of 
the image display unit 28 is ON, various setting states of 
the image display apparatus 400 are indicated by 
means of an image or voice also using the image dis- 
play unit 28. 

[0578] On the other hand, if the mode dial 60 is set at 
an image sensing mode position (step S1 108). a prede- 
termined image sensing process is executed (step 
Si 109). Upon completion of the process, the flow 
returns to step Si 102. 

[0579] If the mode dial 60 is set at any of other mode 
positions (step S1108), the system control circuit 50 



executes a process corresponding to the selected mode 
(step S1 110). Upon completion of the process, the flow 
returns to step S1102. 

[0580] If the mode dial 60 is set at a panoramic mode 
5 position (step S1 1 08), the system control circuit 50 dis- 
plays a panoramic mode selection window on the image 
display unit (step Si 1 1 1 ). 

[0581] The user of the image processing apparatus 
400 can select an arbitrary one of a plurality of pano- 
10 ramie image sensing modes by observing the pano- 
ramic mode selection window displayed on the image 
display unit 28. 

[0582] The panoramic image sensing modes include 
a 2 x 2 panoramic mode, horizontal panoramic mode, 

rs vertical panoramic mode, and the like. In these modes, 
by combining a plurality of sensed images, a panoramic 
image, which is extended two-dimensionally, horizon- 
tally, or vertically, can be obtained. 
[0583] A series of sensed images are appended with 

20 information indicating their positional relationship, and 
can be combined to generate a signal panoramic image 
by the image processing apparatus 400 itself or after 
they are transferred to, e.g., a computer or the like. 
[0584] Note that an example of the panoramic mode 

25 selection window is the same as that shown in Fig. 9 
that has already been described. 
[0585] Referring to Fig. 9. the left icon is used for 
selecting the 2 x 2 panoramic mode for sensing a two- 
dimensional panoramic image, the central icon is used 

30 for selecting the horizontal panoramic mode for sensing 
a horizontal panoramic image, and the right icon is used 
for selecting a vertical panoramic mode for sensing a 
vertical panoramic image. 

[0586] The user of the image processing apparatus 
35 400 can select a desired panoramic mode by the selec- 
tion/change switch 66 (e.g., [+] key), and can determine 
that panoramic mode by the determination/execution 
switch 68 (e.g., [SET] key). 

[0587] When the panoramic mode is selected using 
40 the selection/change switch 66 and determination/exe- 
cution switch 68, the system control circuit 50 sets initial 
values and memory areas of flags, parameters, and var- 
iables, reads out a display window serving as a user 
interface, and so forth in a 2 x 2 panoramic mode initial 
45 setup process (step S1 1 1 3), a vertical panoramic mode 
initial setup process (step S1 114), or a horizontal pano- 
ramic mode initial setup process (step S1 1 15) in corre- 
spondence with the selected panoramic mode (step 
S1112), ancl displays a panoramic image sensing win- 
so dow corresponding to the selected mode on the image 
display unit 28 (step S1 1 16). The flow then advances to 
step S1 131. 

[0588] Note that Figs. 10 A and 10B that have already 
been described show examples of the horizontal pano- 
55 ramie mode image sensing window. Figs. 1 1 A and 1 1B 
that have already been described show examples of the 
vertical panoramic mode image sensing window, and 
Figs. 12A and 12B that have already been described 
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show examples of the 2 x 2 panoramic mode image 
sensing window. 

[0589] Fig. 1 0A shows an image sensing window that 
creates a panoramic window by continuously repeating 
image sensing to the right, and Fig. 10B shows an 
image sensing window that creates a panoramic win- 
dow by continuously repeating image sensing to the left. 
[0590] Referring to Figs. 10A and 10B, a through 
image from the electronic viewfinder is displayed on 
region *T' to sense the first image of a series of pano- 
ramic images. Before completion of image sensing of 
the first image, rightward image sensing (Fig. 10A) or 
leftward image sensing (Fig. 10B) can be desirably 
selected using the selection/change switch 66 (e.g., [+] 
key). 

[0591] The. image sensing direction is determined by 
sensing the first image by pressing the shutter button in 
desired one of the display window states shown in Figs. 
10A and 10B. 

[0592] Note that the user can end image sensing in 
the horizontal panoramic mode by pressing the determi- 
nation/execution switch 68 (e.g., [SET] key). 
[0593] Fig. 1 1 A shows an image sensing window that 
creates a panoramic window by continuously repeating 
image sensing upward, and Fig. 11B shows an image 
sensing window that creates a panoramic window by 
continuously repeating image sensing downward. 
[0594] Referring to Figs. 11 A and 11B, a through 
image from the electronic viewfinder is displayed on 
region "1** to sense the first image of a series of pano- 
ramic images. Before completion of image sensing of 
the first image, upward image sensing (Fig. 11 A) or 
downward image sensing (Fig. 11B) can be desirably 
selected using the selection/change switch 66 (e.g., [+] 
key). 

[0595] The image sensing direction is determined by 
sensing the first image by pressing the shutter button in 
desired one of the display window states shown in Figs. 
1lAand 11B. 

[0596] Note that the user can end image sensing in 
the vertical panoramic mode by pressing the determina- 
tion/execution switch 68 (e.g., [SET] key). 
[0597] Fig. 12A shows an image sensing window for 
the first image upon creating a 2 x 2 panoramic window 
by two-dim ensionally repeating image sensing of a total 
of four images in the vertical and horizontal directions, 
and Fig. 1 2B shows an image sensing window for the 
second image. 

[0598] Referring to Figs. 12A and 12B, a through 
image from the electronic viewfinder is displayed on, 
e.g., an upper right region to sense the first one of pan- 
oramic images. 

[0599] Upon completion of image sensing of the first 
image, a playback image of the sensed image is dis- 
played on. e.g.. the upper right region, and a through 
image from the electronic viewfinder is displayed on, 
e.g., the lower right region to sense the second image. 
[0600] The image sensing region of the electronic 



viewfinder is switched in turn to the next region after 
each image sensing. However, image sensing may be 
started from an arbitrary region, or may be redone by 
returning to the already sensed region. For this pur- 

5 pose, the image sensing region can be desirably shifted 
using the selection/change switch 66 (e.g., [+] switch). 
[0601] The image sensing region to be re-sensed is 
determined and image sensing is executed by pressing 
the shutter button in a display window state that selects 

10 a desired region. 

[0602] Note that the user can end image sensing in 
the 2 x 2 panoramic mode by pressing the determina- 
tion/execution switch 68 (e.g., [SET] key). 
[0603] The system control circuit 50 initializes flags 

is and variables set in its internal memory or the memory 
52. For example, the system control circuit 50 resets a 
panoramic image sensing start flag (step S1 131), sets 
variables m and n, and initializes them to m = 1 and n = 
1 , respectively (steps S1 133 and S1 132). 

20 [0604] Note that the panoramic image sensing start 
flag is reset to indicate that no panoramic image sens- 
ing processes have been done. 

[0605] Also, the variable n indicates the position of the 
current image sensing region in image sensing of each 

25 panoramic mode. 

[0606] The variable m is used for returning the image 
sensing region of interest to the immediately preceding 
region or canceling such return, upon executing image 
sensing again in the horizontal and vertical panoramic 

30 modes. 

[0607] An example of the method of using these flag 
and variables will be explained later. 
[0608] If the mode dial 60 is set at any of mode posi- 
tions other than the panoramic mode (step S1 134). the 

35 system control circuit 50 ends image sensing in the pan- 
oramic mode, and the flow returns to step S1 102. In this 
manner, the user of the image processing apparatus 
400 can end the panoramic mode anytime he or she 
wants, and can select another mode. 

40 [0609] If end of the panoramic mode is selected by 
pressing the determination/execution switch 68 (e.g., 
[SET] key) (step S1 135) while the mode dial 60 is kept 
set at the panoramic mode position (step S1134), i.e., 
the display window of the image display unit 28 keeps 

45 displaying the corresponding panoramic image sensing 
window, a panoramic mode end confirmation message 
is displayed. Furthermore, if confirmation of. end is 
selected (step S1136), the flow returns to step S1 11 1 of 
displaying the panoramic mode selection window. 

so [061 0] The system control circuit 50 checks if the lens 
unit 300 is attached to the image processing apparatus 
400 (step S1161). If the lens unit 300 is attached, the 
flow advances to step S1 137. 

[061 1 ] If the lens unit 300 is not attached to the image 
55 processing apparatus 400 (step S1161). the system 
control circuit 50 checks the status of the panoramic 
image sensing start flag (step S1162). 
[0612] If the panoramic image sensing start flag is 
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reset i.e.. if the first image is not sensed yet in a series 
of panoramic image sensing processes (step S1162). 
the system control circuit 50 displays that no lens is 
attached using the indication unit 54 and/or image dis- 
play unit 28 to call user's attention (step Si 163). The 
flow then advances to step Si 1 37. 
[061 3] If the panoramic image sensing start flag is set, 
i.e., if the first image has already been sensed in a 
series of panoramic image sensing processes (step 
S1 162), the system control circuit 50 makes a predeter- 
mined alert display by means of an image or voice using 
the indication unit 54 and/or the image display unit 28 
(step S1164). Furthermore, if confirmation of the end of 
a series of panoramic image sensing processes is 
selected (step Si 165), the series of panoramic image 
sensing processes executed so far end, and the flow 
returns to step S1 1 31 to start a new series of panoramic 
image sensing processes. 

[0614] If cancel of the end of a series of panoramic 
image sensing processes is selected (step S1 165), the 
flow returns to step S1 134. 

[061 5] With this process, an image, which is unsuita- 
ble for panoramic synthesis, can be prevented from 
being sensed when the lens unit 300 is exchanged by 
another lens unit with a different focal length during a 
series of panoramic image sensing processes. 
[061 6] If the selection/change switch 66 (e.g.. [+] key) 
has been pressed while the image display unit 28 dis- 
plays one of the above-mentioned panoramic image 
sensing windows (step S1137), the system control cir- 
cuit 50 executes a predetermined process in corre- 
spondence with the setups of the panoramic mode (step 
S1138). 

[061 7] If the selection/change switch 66 (e.g., [+] key) 
is not pressed (step Si 137), the flow advances to step 
S1146. 

[0618] If the 2 x 2 panoramic mode is selected (step 
S1138), the system control circuit 50 sets the variable n 
to be n = n + 1 to select the next image sensing region 
(step S1 139). If n = 5 (step S1 140), the first image sens- 
ing region is selected by setting n = n - 4 (step S1 141). 
After that, the flow advances to step S1 146. 
[0619] More specifically, every time the selec- 
tion/change switch 66 (e.g., [+] key) is pressed, the 
image sensing region of interest shifts to one of the four 
image sensing regions in the 2x2 panoramic mode. For 
example, the image sensing region of interest shifts in 
the order of the upper right region, lower right region, 
lower left region, upper left region, upper right region, 
lower right region 

[0620] On the other hand, if the horizontal or vertical 
panoramic mode is selected (step S1138), and if the 
panoramic image sensing start flag is reset, i.e.. if the 
first image is not sensed yet in a series of panoramic 
image sensing processes (step S1142). the system 
control circuit 50 reverses the image sensing direction 
of the selected panoramic mode (step S1143). After 
that, the flow advances to step S1 146. 



[0621] Note that the image sensing direction can be 
repetitively reversed before the first image is sensed. 
[0622] Figs. 10A and 10B that have already been 
described show examples of an image sensing direction 

5 reverse window in the horizontal panoramic mode. 

[0623] Fig. 10A shows a window upon repeating 
image sensing to the right, and Fig. 10B shows a win- 
dow upon repeating image sensing to the left. 
[0624] Figs. 1 1 A and 1 1 B that have already been 

10 described show examples of an image sensing direction 
reverse window in the vertical panoramic mode. 
[0625] Fig. 11A shows a window upon repeating 
image sensing upward, and Fig. 11B shows a window 
upon repeating image sensing downward. 

75 [0626] On the other hand, if the horizontal or vertical 
panoramic mode is selected (step S1138) and if the 
panoramic image sensing start flag is set. i.e., if the first 
image has already been sensed in a series of pano- 
ramic image sensing processes (step Si 142), the sys- 

20 tern control circuit 50 computes m = m x (-1 ), the sign of 
which changes like -1, +1. -1, +1.... in every computa- 
tion (step Si 144), and also computes n = n + m for the 
variable n, which changes like n=n-1.n = n + 1.n = n 
- 1 . n = n + 1 .... in every computation (step Si 145). After 

25 that, the flow advances to step S1 1 46. 

[0627] With this control, movement to the already 
sensed region (which is limited to the one sensed last) 
or cancel that movement can be repetitively selected to 
sense an image again. 

30 [0628] After panoramic image sensing of the n-th 
image region, the user presses the selection/change 
switch 66 (e.g., [+] key) to confirm if quick review images 
of the (n-1)-th and n-th image regions smoothly join at 
their boundary regions. Then, the user temporarily 

35 releases the selection/change switch 66 (e.g., [+] key) to 
display a through image on the n-th image region, and 
presses the selection/change switch 66 (e.g.. [+] key) 
once again to display the original quick review image on 
the n-th image region and a through image on the (n+1)- 

40 th image region, thus executing the next panoramic 
image sensing. 

[0629] After panoramic image sensing of the n-th 
image region, upon pressing the selection/change 
switch 66 (e.g., [+] key) to confirm if quick review images 

45 of the (n-1)-th and n-th image regions smoothly join at 
their boundary regions, if the user determines that 
image sensing must be redone, he or she temporarily 
releases the.selecti on/change switch 66 (e.g.. [+] key) to 
display a through image on the n-th image region, and 

so presses the shutter switch SW2 in that window state to 
replace the image with the newly sensed image. Then, 
the user can execute the next panoramic image sens- 
ing. 

[0630] The system control circuit 50 selects the n-th 
55 image region (step S1 146). If the window layout must be 
changed in correspondence with the selected image 
region (step S1147), the system control circuit 50 
switches and displays a new window layout (step 
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Si 1 48), and the flow then advances to step S1 1 91 . 
[0631] In this fashion, a window corresponding to the 
reversed image sensing direction, as described above, 
a window corresponding to the changed image sensing 
region, a window that has updated characters and 
images for comments, or the like can be displayed as 
needed. 

[0632] Upon sensing the second and subsequent 
images, the image sensing regions that display already 
sensed images are set in a quick review display state, 
and the image sensing region selected for the next 
image sensing is set in a through display state, thus dis- 
playing a playback image of the already sensed image 
and the next image to be sensed to partially overlap 
ach other. In this manner, the user of the image 
processing apparatus 400 can easily determine the next 
image sensing angle. 

[0633] The system control circuit 50 checks if the lens 
unit 300 is attached to the image processing apparatus 
400 (step S1191). If the lens unit 300 is attached, the 
flow advances to step S1 1 71 . 

[0634] If the lens unit 300 is not attached to the image 
processing apparatus 400 (step S1191), the system 
control circuit 50 displays that no lens is attached using 
the indication unit 54 and/or image display unit 28 to call 
user's attention (step S1192). The flow then returns to 
step S1134. 

[0635] With this process, an image, which is unsuita- 
ble for panoramic synthesis, can be prevented from 
being sensed when the lens unit 300 is exchanged by 
another lens unit with a different focal length during a 
series of panoramic image sensing processes. 
[0636] The system control circuit 50 sets a through 
display state in which sensed image data is displayed 
as it is sensed (step S1171), and the flow advances to 
step Si 172. 

[0637] In through display, data, which have been writ- 
ten in the image display memory 24 via the image sens- 
ing element 14, A/D converter 16, image processing 
circuit 20, and memory control circuit 22, are displayed 
on the aforementioned image sensing region of the 
image display unit 28 via the memory control circuit 22 
and D/A converter 26, thereby implementing an elec- 
tronic viewf inder function. 

[0638] If the shutter switch SW1 is not pressed (step 
S1 1 72), the flow returns to step S1 134. 
[0639] If the shutter switch SW1 has been pressed 
(step S1 1 72), the system control circuit 50 sets the dis- 
play state of the image display unit 28 in a freeze display 
state (step S1173), and the flow advances to step 
S1174. 

[0640] In freeze display, image data on the image dis- 
play memory 24 is inhibited from being rewritten via the 
image sensing element 14. A/D converter 16, image 
processing circuit 20. and memory control circuit 22. 
and the latest written image data is displayed on the 
above-mentioned image display region of the image dis- 
play unit 28 via the memory control circuit 22 and D/A 



converter 26, thus displaying a frozen image on the 
electronic viewfinder. 

[0641 ] The system control circuit 50 executes distance 
measurement to adjust the focus of the photographing 
5 lens 310 on the object, and executes photometry to 
determine the aperture value and shutter speed (step 
S1 174). In photometry, the electronic flash is set if nec- 
essary. 

[0642] The details of the flow of this distance meas- 
io urement/photometry process step S1 1 74 are the same 
as those of the second embodiment that has been 
described using Fig. 5. 

[0643] Upon completion of the distance measure- 
ment/photometry process step S1 1 74, the system con- 
15 trol circuit 50 sets the display state of the image display 
unit 28 in-a through display state (step S1175). and the 
flow advances to step S1 1 76. Note that the through dis- 
play state in step S1 1 75 is the same as that in step 
S1171. 

20 [0644] If the shutter switch SW2 is not pressed (step 
Si 176) and the shutter switch SW1 is canceled (step 
Si 1 77). the flow returns to step S1 1 34. 
[0645] If the shutter switch SW2 has been pressed 
(step Si 176). the system control circuit 50 sets the dis- 

25 play state of the image display unit 28 in a fixed-color 
display state (step S1178). and the flow advances to 
step S1 179. 

[0646] In fixed-color display, in place of sensed image 
data, which has been written in the image display mem- 
30 ory 24 via the image sensing element 1 4. A/D converter 
16, image processing circuit 20, and memory control 
circuit 22. substitute fixed-color image data is displayed 
on the image display unit 28 via the memory control cir- 
cuit 22 and D/A converter 26, thus displaying a fixed - 
35 color image on the electronic viewfinder. 

[0647] The system control circuit 50 executes an 
image sensing process including an exposure process 
for writing sensed image data in the memory 30 via the 
image sensing element 14, A/D converter 16. image 
40 processing circuit 20, and memory control circuit 22 or 
directly from the A/D converter 16 via the memory con- 
trol circuit 22, and a development process for executing 
various processes by reading out image data written in 
the memory 30 using the memory control circuit 22 and 
45 image processing circuit 20 if required (step S1 1 79). 
[0648] The details of the flow of this image sensing 
process step S1 179 are the same as those of the sec- 
ond embodiment that has been described using Fig. 6. 
[0649] The system control circuit 50 executes a dis- 
so play image process for reading out image data written in 
the memory 30 by the image sensing process in step 
S1 179, and transferring the readout data to the image 
display memory 24 via the memory control circuit 22 
(step S1180). 

55 [0650] The details of the flow of this display image 
process step S1 180 are the same as those of the sec- 
ond embodiment that has been described using Fig. 7. 
[0651] The system control circuit 50 sets the display 
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state of the image display unit 28 in a quick review dis- 
play state (step S1 181), and the flow advances to step 
S1182. 

[0652] In quick review display, image data transferred 
to the image display memory 24 by the display image 
process in step S1180 is displayed on the aforemen- 
tioned image sensing region of the image display unit 
28 via the memory control circuit 22 and D/A converter 
26, thus implementing an electronic viewfinder function 
that automatically plays back a sensed image. 
[0653] The system control circuit 50 executes a com- 
pression process in which various image processes are 
done using the memory control circuit 22 and image 
processing circuit 20 if required by reading out sensed 
image data written in the memory 30, or an image is 
compressed in correspondence with the selected mode 
using the compression/expansion circuit 32 (step 
S1182). After that, the system control circuit 50 exe- 
cutes a recording process for writing image data in the 
recording medium 500 or 510 such as a memory card, 
compact flash card, or the like via the interface 90 or 94, 
and the connector 92 or 96 (step S1 183). 
[0654] The details of the flow of this compression 
process step S1 182 are the same as those in the sec- 
ond embodiment that has already been described using 
Fig. 8. 

[0655] When the image display unit 28 is ON, a mes- 
sage such as "BUSY" or the like that indicates that a 
write is in progress is displayed on the image display 
unit 28 during a write of image data on the recording 
medium 500 or 510. 

[0656] Furthermore, the indication unit 54 makes a 
recording medium write access indication by flickering 
an LED or the like in combination with the above mes- 
sage. 

[0657] The system control circuit 50 continues quick 
review display on the image display unit 28 until the 
shutter switch SW2 is released (step S1184). With this 
display, the user can carefully check the sensed image 
while pressing the shutter switch SW2. 
[0658] If the shutter switch SW2 is released (step 
S1184), the system control circuit 50 sets the pano- 
ramic image sensing start flag (step S1 185) to indicate 
that the first image has already been sensed in a series 
of panoramic image sensing processes. In this manner, 
as has been described in step S1 142, the image sens- 
ing direction can be inhibited from being reversed after 
the first image is sensed in the horizontal or vertical 
panoramic mode. 

[0659] The system control circuit 50 sets the variable 
n to be n ss n + 1 to select the next image sensing region 
(step Si 186). and resets the variable m to m = 1 (step 
S1 187). Then, the flow returns to step S1 134 to repeat 
a series of panoramic image sensing processes. 
[0660] In this manner, after the control returns to step 
S1 134 upon releasing the shutter switch SW2, the next 
image sensing region is selected in step S1146, the 
image sensing regions which include the previous 



image sensing region and display sensed images are 
set in the quick review state, and the next image sensing 
region is set in the through display state, so as to display 
the sensed images and the next image to be sensed to 

5 overlap each other in step S1 1 48, thus allowing the user 
to easily sense panoramic images. 
[0661] Figs. 14A and 14B that have already been 
described show examples of the image sensing window 
before and after completion of sensing of the n-th image 

10 in the horizontal panoramic mode. 

[0662] Fig. 14A shows the image sensing window 
before sensing the first image, and Fig. 14B shows the 
image sensing window upon releasing the shutter 
switch SW2 after the first image is sensed. Although no 

is dedicated explanatory view is available, before the shut- 
ter switch SW2 is released after the first image sensing, 
the first image sensing region in Fig. 14A on the image 
sensing window is set in a quick review display state. 
[0663] Figs. 15A and 15B that have already been 

20 described show examples of the image sensing window 
before and after completion of sensing of the n-th image 
in the vertical panoramic mode. 

[0664] Fig. 15A shows the image sensing window 
before sensing the first image, and Fig. 15B shows the 

25 image sensing window upon releasing the shutter 
switch SW2 after the first image is sensed. Although no 
dedicated explanatory view is available, before the shut- 
ter switch SW2 is released after the first image sensing, 
the first image sensing region in Fig. 15A on the image 

30 sensing window is set in a quick review display state. 
[0665] Furthermore. Figs. 12A to 13B that have 
already been described show examples of the image 
sensing window before and after completion of sensing 
of the n-th image in the 2 x 2 panoramic mode. 

35 [0666] Fig. 12A shows the image sensing window 
before the first image is sensed. Fig. 12B shows the 
image sensing window upon releasing the shutter 
switch SW2 after the first image is sensed, i.e., before 
the second image is sensed. 

40 [0667] Fig. 13A shows the image sensing window 
upon releasing the shutter switch SW2 after the second 
image is sensed, i.e., before the third image is sensed, 
and Fig. 13B shows the image sensing window upon 
canceling the shutter switch SW2 after the third image is 

45 sensed, i.e., before the fourth image is sensed. 

[0668] Although no dedicated explanatory view is 
available, before the shutter switch SW2 is released 
after the n-th image sensing, the n-th image sensing 
region in Figs^. 12A to 13B on the image sensing window 

50 is set in a quick review display state. When the shutter 
switch SW2 is released, a through display of the (n+1)- 
th image sensing region as the next region is added. 
[0669] As described above, according to the fourth 
embodiment, after a series of panoramic image sensing 

55 processes have started, a predetermined alert is gener- 
ated upon detection of the absence of the lens unit 300, 
and the panoramic image sensing processes are tem- 
porarily interrupted to start a new series of panoramic 
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image sensing processes. In this way, an image, which 
is unsuitable for panoramic synthesis, can be prevented 
from being sensed when the lens unit 300 is exchanged 
by another lens unit with a different focal length during a 
series of panoramic image sensing processes. 5 

(Fifth Embodiment) 

[0670] Fig. 3 1 is a block diagram showing the arrange- 
ment of an image processing apparatus according to io 
the fifth embodiment of the present invention. 
[0671 ] In Fig. 3 1 , only a difference from Fig. 1 6 will be 
explained. 

[0672] Reference numeral 426 denotes a lens attach- 
ment/detachment switch used upon attaching/detach- 15 
ing the lens unit 300 to/from the image processing 
apparatus 400. 

[0673] The operation of the fifth embodiment will be 
explained below with reference to Figs. 32. 33, 34, 45, 
36, 5. 6, 7, and 8. Note that the processes shown in so 
Figs. 5, 6, 7, and 8 are the same as those in the second 
embodiment, and a detailed description thereof will be 
omitted. 

[0674] The operation of the image processing appara- 
tus 400 will be described with the aid of Figs. 32, 33. 34, 25 
35. and 36. 

[0675] Upon power ON after battery exchange or the 
like, the system control circuit 50 initializes flags, control 
variables, and the like, and initializes the respective 
units of the image processing apparatus 400 (step 30 
S1601). 

J0676] The system control circuit 50 checks the setting 
position of the mode dial 60. If the mode dial 60 is set at 
a power OFF position (step S1602), the system control 
circuit 50 executes a predetermined end process (step 35 
Si 603). More specifically, the system control circuit 50 
changes the indications of the respective indication 
units to an end state, protects an image sensing unit by 
closing a barrier if the lens unit 300 comprises the bar- 
rier or the like serving as a protection means, records 40 
required parameters and setting values including flags, 
control variables, and the like, and the setting mode in 
the nonvolatile memory 56, cuts off unnecessary power 
supply to the respective units of the image processing 
apparatus 400 including the image display unit 28 by the 45 
power supply control means 80, and so forth. After that 
the flow returns to step S1602. 

[0677] If the mode dial 60 is set at any of other mode 
positions (step Si 602), the system control circuit 50 
checks using the power supply control means 80 if the so 
remaining capacity and operation state of the power 
supply means 86 comprising batteries and the like pose 
any problem in the operation of the image processing 
apparatus 400 (step S1604). If any problem is found, a 
predetermined alert indication is made by means of an 55 
image or voice using the indication unit 54 (step S1 606). 
and the flow then returns to step Si 602. 
[0678] When the image display of the image display 



unit 28 is ON, a predetermined alert display is made by 
means of an image or voice also using the image dis- 
play unit 28. 

[0679] If no problem is found in the power supply 
means 86 (step S1604), the system control circuit 50 
checks if the operation state of the recording medium 
500 or 510 poses any problem in the operation of the 
image processing apparatus 400, in particular, record- 
ing/playback of image data to/from the recording 
medium (step S1605). If any problem is found, a prede- 
termined alert indication is made by means of an image 
or voice using the indication unit 54 (step S1606), and 
the flow then returns to step S1602. 
[0680] When the image display of the image display 
unit 28 is ON, a predetermined alert display is made by 
means of an image or voice also using the image dis- 
play unit 28. 

[0681] If no problem is found in the operation state of 
the recording medium 500 or 510 (step S1605), various 
setting states of the image display apparatus 400 are 
indicated by means of an image or voice using the indi- 
cation unit 54 (step S1607). When the image display of 
the image display unit 28 is ON, various setting states of 
the image display apparatus 400 are indicated by 
means of an image or voice also using the image dis- 
play unit 28. 

[0682] On the other hand, if the mode dial 60 is set at 
an image sensing mode position (step S1608). a prede- 
termined image sensing process is executed (step 
S1609). Upon completion of the process, the flow 
returns to step S1602. 

[0683] If the mode dial 60 is set at any of other mode 
positions (step S1608), the system control circuit 50 
executes a process corresponding to the selected mode 
(step S1610). Upon completion of the process, the flow 
returns to step S1602. 

[0684] If the mode dial 60 is set at a panoramic mode 
position (step S1608), the system control circuit 50 dis- 
plays a panoramic mode selection window on the image 
display unit (step Si 61 1). 

[0685] The user of the image processing apparatus 
400 can select an arbitrary one of a plurality of pano- 
ramic image sensing modes by observing the pano- 
ramic mode selection window displayed on the image 
display unit 28. 

[0686] The panoramic image sensing modes include 
a 2 x 2 panoramic mode, horizontal panoramic mode, 
vertical panoramic mode, and the like. In these modes, 
by combining a-plurality of sensed images, a panoramic 
image, which is extended two-dimensionally, horizon- 
tally, or vertically, can be obtained. 
[0687] A series of sensed images are appended with 
information indicating their positional relationship, and 
can be combined to generate a signal panoramic image 
by the image processing apparatus 400 itself or after 
they are transferred to, e.g., a computer or the like. 
[0688] Note that an example of the panoramic mode 
selection window is the same as that shown in Fig. 9 



VJSDOCID: <EP 0942598A2_I_> 



44 



87 



EP0 942 598 A2 



88 



that has already been explained. 
[0689] Referring to Fig. 9. the left icon is used for 
selecting the 2x2 panoramic mode for sensing a two- 
dimensional panoramic image, the central icon is used 
for selecting the horizontal panoramic mode for sensing 
a horizontal panoramic image, and the right icon is used 
for selecting a vertical panoramic mode for sensing a 
vertical panoramic image. 

[0690] The user of the image processing apparatus 
400 can select a desired panoramic mode by the selec- 
tion/change switch 66 (e.g., [+] key), and can determine 
that panoramic mode by the determination/execution 
switch 68 (e.g., [SET] key). 

[0691 ] When the panoramic mode is selected using 
the selection/change switch 66 and determination/exe- 
cution switch 68, the system control circuit 50 sets initial 
values and memory areas of flags, parameters, and var- 
iables, reads out a display window serving as a user 
interface, and so forth in a 2 x 2 panoramic mode initial 
setup process (step S1613), a vertical panoramic mode 
initial setup process (step Si 61 4), or a horizontal pano- 
ramic mode initial setup process (step Si 61 5) in corre- 
spondence with the selected panoramic mode (step 
Si 612). and displays a panoramic image sensing win- 
dow corresponding to the selected mode on the image 
display unit 28 (step S1 61 6). The flow then advances to 
step S1631. 

[0692] Note that Figs. 10A and 10B that have already 
been explained show examples of the horizontal pano- 
ramic mode image sensing window, Figs. 1 1 A and 1 1 B 
show examples of the vertical panoramic mode image 
sensing window, and Figs. 12A and 12B that have 
already been explained show examples of the 2 x 2 
panoramic mode image sensing window. 
[0693] Fig. 10 A shows an image sensing window that 
creates a panoramic window by continuously repeating 
image sensing to the right, and Fig. 10B shows an 
image sensing window that creates a panoramic win- 
dow by continuously repeating image sensing to the left. 
[0694] Referring to Figs. 10A and 10B, a through 
image from the electronic viewfinder is displayed on 
region "1" to sense the first image of a series of pano- 
ramic images. Before completion of image sensing of 
the first image, rightward image sensing (Fig. 10A) or 
leftward image sensing (Fig. 10B) can be desirably 
selected using the selection/change switch 66 (e.g., [+] 
key). 

[0695] The image sensing direction is determined by 
sensing the first image by pressing the shutter button in 
desired one of the display window states shown in Figs. 
10Aand 10B. 

[0696] Note that the user can end image sensing in 
the horizontal panoramic mode by pressing the determi- 
nation/execution switch 68 (e.g., [SET] key). 
[0697] Fig. 1 1 A shows an image sensing window that 
creates a panoramic window by continuously repeating 
image sensing upward, and Fig. 11B shows an image 
sensing window that creates a panoramic window by 



continuously repeating image sensing downward. 
[0698] Referring to Figs. 11A and 11B, a through 
image from the electronic viewfinder is displayed on 
region "1" to sense the first image of a series of pano- 
5 ramie images. Before completion of image sensing of 
the first image, upward image sensing (Fig. 11 A) or 
downward image sensing (Fig. 1 1B) can be desirably 
selected using the selection/change switch 66 (e.g., [+] 
key). 

10 [0699] The image sensing direction is determined by 
sensing the first image by pressing the shutter button in 
desired one of the display window states shown in Figs. 
11Aand 11B. 

[0700] Note that the user can end image sensing in 
is the vertical panoramic mode by pressing the determina- 
tion/execution switch 68 (e.g., [SET] key). 
[0701] Fig. 12A shows an image sensing window for 
the first image upon creating a 2 x 2 panoramic window 
by two-dimensionally repeating image sensing of a total 
20 of four images in the vertical and horizontal directions, 
and Fig. 12B shows an image sensing window for the 
second image. 

[0702] Referring to Figs. 12A and 12B, a through 
image from the electronic viewfinder is displayed on. 
25 e.g., an upper right region to sense the first one of pan- 
oramic images. 

[0703] Upon completion of image sensing of the first 
image, a playback image of the sensed image is dis- 
played on, e.g.. the upper right region, and a through 
30 image from the electronic viewfinder is displayed on, 
e.g., the lower right region to sense the second image. 
[0704] The image sensing region of the electronic 
viewfinder is switched in turn to the next region after 
each image sensing. However, image sensing may be 
35 started from an arbitrary region, or may be redone by 
returning to the already sensed region. For this pur- 
pose, the image sensing region can be desirably shifted 
using the selection/change switch 66 (e.g., [+] switch). 
[0705] The image sensing region to be re-sensed is 
40 determined and image sensing is executed by pressing 
the shutter button in a display window state that selects 
a desired region. 

[0706] Note that the user can end image sensing in 
the 2 x 2 panoramic mode by pressing the determina- 
45 tion/execution switch 68 (e.g., [SET] key). 

[0707] The system control circuit 50 initializes flags 
and variables set in its internal memory or the memory 
52. For example, the system control circuit 50 resets a 
panoramic kpage sensing start flag (step S1631), sets 
so variables m and n, and initializes them to m = 1 and n = 
1, respectively (steps S1633 and S1632). 
[0708] Note that the panoramic image sensing start 
flag is reset to indicate that no panoramic image sens- 
ing processes have been done. 
55 [0709] Also, the variable n indicates the position of the 
current image sensing region in image sensing of each 
panoramic mode. 

[0710] The variable m is used for returning the image 
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sensing region of interest to the immediately preceding 
region or canceling such return, upon executing image 
sensing again in the horizontal and vertical panoramic 
modes. 

[0711] An example of the method of using these flag s 
and variables will be explained later. 
[071 2] If the mode dial 60 is set at any of mode posi- 
tions other than the panoramic mode (step S1634), the 
system control circuit 50 ends image sensing in the pan- 
oramic mode, and the flow returns to step S1.602. In this 10 
manner, the user of the image processing apparatus 
400 can end the panoramic mode anytime he or she 
wants, and can select another mode. 
[071 3] If end of the panoramic mode is selected by 
pressing the determination/execution switch 68 (e.g., is 
[SET] key) (step S1635) while the mode dial 60 is kept 
set at the panoramic mode position (step S1634), i.e., 
the display window of the image display unit 28 keeps 
displaying the corresponding panoramic image sensing 
window, a panoramic mode end confirmation message 20 
is displayed. Furthermore, if confirmation of end is 
selected (step S1636), the flow returns to step S161 1 of 
displaying the panoramic mode selection window. 
[0714] The system control circuit 50 detects if opera- 
tion for detaching the lens unit 300 using the lens 25 
attachment/detachment switch 426 is about to start 
(step S1661). If operation for detaching the lens unit 
300 is not about to start, the flow advances to step 
S1637. 

[0715] If operation for detaching the lens unit 300 is 30 
about to start (step S1661), the system control circuit 50 
checks the status of the panoramic image sensing start 
flag (step S1 662). 

[071 6] If the panoramic image sensing start flag is 
reset, i.e., if the first image is not sensed yet in a series 35 
of panoramic image sensing processes (step S1662), 
the system control circuit 50 saves various kinds of 
information such as the setting mode, setting values, 
and the like, which were used in the respective units of 
the image processing apparatus 400 while the lens unit 40 
300 is attached to the apparatus 400, in a nonvolatile 
memory in the lens system control circuit 350 via the 
interface 420, connector 422, connector 322, and inter- 
face 320 (step S1663), and permits detachment of the 
lens unit 300 (step Si 664). After that, the flow advances 45 
to step S1 637. 

[071 7] Note that a method of permitting detachment of 
the lens unit 300 may use any of a method of unlocking 
a mechanism such as an electromagnetic lock that 
inhibits the lens unit from being detached, a method of so 
removing a cover of a detachment button of the lens unit 
300, and the like. 

[0718] Various kinds of information saved in the non- 
volatile memory in the lens system control circuit 350 
can be read and used again when the detached lens 55 
unit 300 is attached to the image processing apparatus 
400 again. With this control, the user need not repeat 
various setups in correspondence with the lens unit 300 



used every time he or she exchanges the lens unit 300. 
resulting in convenience. 

[071 9] If the panoramic image sensing start flag is set, 
i.e., if the first image has already been sensed in a 
series of panoramic image sensing processes (step 
S1662). the system control circuit 50 makes a predeter- 
mined alert display by means of an image or voice using 
the indication unit 54 and/or the image display unit 28 
(step S1665). Furthermore, if confirmation of the end of 
a series of panoramic image sensing processes is 
selected (step Si 666), the system control circuit 50 
temporarily ends the series of panoramic image sensing 
processes executed so far, and the flow returns to step 
S1631 to start a new series of panoramic image sens- 
ing processes. 

[0720] On the other hand, rf cancel of the end of a 
series of panoramic image sensing processes is 
selected (step S1666), the flow advances to step 
S1637. 

[0721] In this way, since the lens unit 300 is inhibited 
from being exchanged by another lens unit with a differ- 
ent focal length during a series of panoramic image 
sensing processes, an image, which is unsuitable for 
panoramic synthesis, can be prevented from being 
sensed. 

[0722] If the selection/change switch 66 (e.g.. [+] key) 
has been pressed while the image display unit 28 dis- 
plays one of the above-mentioned panoramic image 
sensing windows (step S1637), the system control cir- 
cuit 50 executes a predetermined process in corre- 
spondence with the setups of the panoramic mode (step 
SI 638). 

[0723] If the selection/change switch 66 (e.g.. [+] key) 
is not pressed (step S1637), the flow advances to step 
S1646. 

[0724] If the 2 x 2 panoramic mode is selected (step 
S1 638), the system control circuit 50 sets the variable n 
to be n ■ n + 1 to select the next image sensing region 
(step S1639). If n m 5 (step S1640), thefirst image sens- 
ing region is selected by setting n « n - 4 (step Si 641). 
After that, the flow advances to step S1646. 
[0725] More specifically, every time the selec- 
tion/change switch 66 (e.g., [+] key) is pressed, the 
image sensing region of interest shifts to one of the four 
image sensing regions in the 2 x 2 panoramic mode. For 
example, the image sensing region of interest shifts in 
the order of the upper right region, lower right region, 
lower left region, upper left region, upper right region, 
lower right region 

[0726] On the other hand, if the horizontal or vertical 
panoramic mode is selected (step S1638). and if the 
panoramic image sensing start flag is reset, i.e.. if the 
first image is not sensed yet in a series of panoramic 
image sensing processes (step S1642). the system 
control circuit 50 reverses the image sensing direction 
of the selected panoramic mode (step SI 643). After 
that, the flow advances to step S1646. 
[0727] Note that the image sensing direction can be 
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repetitively reversed before the first image is sensed. 
[0728] Figs. 10A and 10B that have already been 
described show examples of an image sensing direction 
reverse window in the horizontal panoramic mode. 
10729] Fig. 10A shows a window upon repeating 
image sensing to the right and Fig. 10B shows a win- 
dow upon repeating image sensing to the left 
[0730] Figs. 11A and 11B that have already been 
described show examples of an image sensing direction 
reverse window in the vertical panoramic mode. 
[0731] Fig. 11A shows a window upon repeating 
image sensing upward, and Fig. 1 1B shows a window 
upon repeating image sensing downward. 
[0732] On the other hand, if the horizontal or vertical 
panoramic mode is selected (step S1638) and if the 
panoramic image sensing start flag is set, i.e.. if the first 
image has already been sensed in a series of pano- 
ramic image sensing processes (step S1642), the sys- 
tem control circuit 50 computes m = m x (-1), the sign of 
wtiich changes like -1, +1, -1, +1,... in every computa- 
tion (step Si 644), and also computes n = n + m for the 
vanaole n. which changes like n = n-1,n = n+1,n = n 
- i . n = n + 1 .... in every computation (step S1645). After 
that, the flow advances to step S1646. 
[0733] With this control, movement to the already 
sensed region (which is limited to the lastly sensed one) 
or cancel that movement can be repetitively selected to 
sense an image again. 

[0734] After panoramic image sensing of the n-th 
image region, the user presses the selection/change 
switch 66 (e.g., [+] key) to confirm if quick review images 
of the (n-l)-th and n-th image regions smoothly join at 
their boundary regions. Then, the user temporarily 
releases the selection/change switch 66 (eg., [+] key) to 
display a through image on the n-th image region, and 
presses the selection/change switch 66 (e.g., [+] key) 
once again to display the original quick review image on 
the n-th image region and a through image on the (n+1)- 
th image region, thus executing the next panoramic 
image sensing. 

[0735] After panoramic image sensing of the n-th 
image region, upon pressing the selection/change 
switch 66 (e.g., [+] key) to confirm if quick review images 
of the (n-1)-th and n-th image regions smoothly join at 
their boundary regions, if the user determines that 
image sensing must be redone, he or she temporarily 
releases the selection/change switch 66 (e.g.. [+] key) to 
display a through image on the n-th image region, and 
presses the shutter switch SW2 in that window state to 
replace the image with the newly sensed image. Then, 
the user can execute the next panoramic image sens- 
ing. 

[0736] The system control circuit 50 selects the n-th 
image region (step S1646). If the window layout must be 
changed in correspondence with the selected image 
region (step S1647), the system control circuit 50 
switches and displays a new window layout (step 
S1 648) The flow then advances to step S1691 . 



[073*7] In this fashion, a window corresponding to the 
reversed image sensing direction, as described above, 
a window corresponding to the changed image sensing 
region, a window that has updated characters and 
5 images for comments, or the like can be displayed as 
needed. 

[0738] Upon sensing the second and subsequent 
images, the image sensing regions that display the 
already sensed images are set in a quick review display 
10 state, and the image sensing region selected for the 
next image sensing is set in a through display state, 
thus displaying a playback image of the already sensed 
image and the next image to be sensed to partially over- 
lap each other. In this manner, the user of the image 
is processing apparatus 400 can easily determine the next 
image sensing angle. 

[0739] The system control circuit 50 checks if the lens 
unit 300 is attached to the image processing apparatus 
400 (step S1691). If the lens unit 300 is attached, the 
flow advances to step S1693. 

[0740] On the other hand, rf the lens unit 300 is not 
attached to the image processing apparatus 400 (step 
S1691), display indicating that no lens is attached is 
made using the indication unit 54 and/or the image dis- 
play unit 28 to call user's attention (step S1692). The 
flow then returns to step S1634. 
[0741 ] In this manner, since image sensing without the 
lens unit 300 is inhibited during a series of panoramic 
image sensing processes, an image, which is unsuita- 
ble for panoramic synthesis, can be prevented from 
being sensed. 

[0742] If the system control circuit 50 determines that 
a new lens unit 300 is attached and various kinds of set- 
ting information saved in the lens unit 300 must be read 
(step S1693). it reads identification information such as 
a number unique to the lens unit 300 or the like, man- 
agement information, function information such as a 
full -open aperture value and minimum aperture value, 
focal length, and the like, and the current and previous 
setting values, which are saved in the nonvolatile mem- 
ory in the lens system control circuit 350, and/or various 
kinds of information such as the setting mode, setting 
values, and the like, which were used in the respective 
units of the image processing apparatus while the lens 
unit 300 was attached to the image processing appara- 
tus 400 previously, onto its internal registers and/or the 
memory 52 via the interface 320. connector 322, con- 
nector 422, and interface 420 (step Si 694). 
[0743] Note that the saved various setting values 
include, for example, the zoom focal length, the opera- 
tion mode used such as a shutter speed priority AE 
mode or aperture priority AE mode, sports image sens- 
ing mode, landscape image sensing mode, panoramic 
image sensing mode, and the like, the ON/OFF state of 
a camera shake prevention function, and the like. 
[0744] The system control circuit 50 sets various cor- 
responding modes suitable for image sensing using the 
attached lens unit 300 on the basis of various kinds of 
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read setting information (step S1695), and the flow then 
advances to step S1 671 . 

[0745] Note that various corresponding modes are set 
as follows. For example, when the lens unit 300 is a tel- 
ephoto lens or telephoto zoom lens, the contents of the 
operation mode display list of the image processing 
apparatus 400 are limited to only the shutter speed pri- 
ority AE mode, sports image sensing mode, and the 
like; the shutter speed priority AE mode, sports image 
sensing mode, and the like are displayed at easy-to- 
select positions; one of the shutter speed priority AE 
mode, sports image sensing mode, and the like is set as 
a default mode; and so forth. 

[0746] On the other hand, when the lens unit 300 is a 
wide-angle lens or wide-angle zoom lens, the contents 
of the operation mode display list of the image process- 
ing apparatus 400 are limited to only the aperture prior- 
ity AE mode, landscape image sensing mode, 
panoramic image sensing mode, and the like; the aper- 
ture priority AE mode, landscape image sensing mode, 
panoramic image sensing mode, and the like are dis- 
played at easy-to-select positions; one of the aperture 
priority AE mode, landscape image sensing mode, pan- 
oramic image sensing mode, and the like is set as a 
default mode; and so forth. 

[0747] Furthermore, since various kinds of information 
saved in the nonvolatile memory in the lens system con- 
trol circuit 350 upon detachment of the lens unit 300 can 
be read and used when the detached lens unit 300 is 
attached to the image processing apparatus 400 again, 
the user need not repeat various setups in correspond- 
ence with the lens unit 300 used every time he or she 
exchanges the lens unit 300, resulting in convenience. 
[0748] On the other hand, some or all of various kinds 
of information saved in the nonvolatile memory in the 
lens system control circuit 350 upon detachment of the 
lens unit 300 may not be used in correspondence with 
the date, day of the week, elapsed time, user informa- 
tion, and the like upon attachment of the detached lens 
unit 300 to the image processing apparatus 400 again. 
With this control, when the lens unit is exchanged fre- 
quently, the need for repeating various setups can be 
obviated. Also, when previous setups have no signifi- 
cance upon exchanging the lens unit after several days, 
default values may be set. 

[0749] Moreover, when user information saved in the 
nonvolatile memory in the lens system control circuit 
350 is used, the need for repeating various setups cor- 
r sponding to the lens unit 300 in units of users of the 
image processing apparatus 400 can be obviated. 
[0750] If the system control circuit 50 determines that 
lens unit 300 remains attached and various setups 
saved in the lens unit 300 need not be read (step 
S1693), the flow advances to step S1671. 
[0751] The system control circuit 50 sets a through 
display state in which sensed image data is displayed 
as it is sensed (step S1671), and the flow advances to 
step S1672. 



[0752] In through display, data which have been writ- 
ten in the image display memory 24 via the image sens- 
ing element 14, A/D converter 16, image processing 
circuit 20, and memory control circuit 22, are displayed 
5 on the aforementioned image sensing region of the 
image display unit 28 via the memory control circuit 22 
and D/A converter 26, thereby implementing an elec- 
tronic viewfinder function. 

[0753] If the shutter SW1 is not pressed (step SI 672), 

w the flow returns to step S1634. 

[0754] If the shutter switch SW1 has been pressed 
(step S1672), the system control circuit 50 sets the dis- 
play state of the image display unit 28 in a freeze display 
state (step S1673), and the flow advances to step 

is Si 674. 

[0755] In freeze display, image data on the image dis- 
play memory 24 is inhibited from being rewritten via the 
image sensing element 14, A/D converter 16, image 
processing circuit 20, and memory control circuit 22. 

20 and the latest written image data is displayed on the 
above-mentioned image display region of the image dis- 
play unit 28 via the memory control circuit 22 and D/A 
converter 26. thus displaying a frozen image on the 
electronic viewfinder. 

25 [0756] The system control circuit 50 executes distance 
measurement to adjust the focus of the photographing 
lens 310 on the object, and executes photometry to 
determine the aperture value and shutter speed (step 
Si 674). In photometry, the electronic flash is set if nec- 

30 essary. 

[0757] The details of the flow of this distance meas- 
urement/photometry process step S1674 are the same 
as those in the second embodiment that has already 
been described using Fig. 5. 

35 [0758] Upon completion of the distance measure- 
ment/photometry process step S1674, the system con- 
trol circuit 50 sets the display state of the image display 
unit 28 in a through display state (step S1675), and the 
flow advances to step Si 676. Note that the through dis- 

40 play state in step Si 675 is the same as that in step 
S1671. 

[0759] If the shutter switch SW2 is not pressed (step 

51676) and the shutter switch SW1 is canceled (step 

51 677) , the flow returns to step S1634. 

45 [0760] If the shutter switch SW2 has been pressed 
(step S1676), the system control circuit 50 sets the dis- 
play state of the image display unit 28 in a fixed -color 
display state (step S1678), and the flow advances to 
step S1 679. 

so [0761 ] In fixed-color display, in place of sensed image 
data, which has been written in the image display mem- 
ory 24 via the image sensing element 14, A/D converter 
16, image processing circuit 20. and memory control 
circuit 22. substitute fixed-color image data is displayed 

55 on the image display unit 28 via the memory control cir- 
cuit 22 and D/A converter 26, thus displaying a fixed- 
color image on the electronic viewfinder. 
[0762] The system control circuit 50 executes an 
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image sensing process including an exposure process 
for writing sensed image data in the memory 30 via the 
image sensing element 14, A/D converter 16. image 
processing circuit 20. and memory control circuit 22 or 
directly from the A/D converter 16 via the memory con- 
trol circuit 22, and a development process for executing 
various processes by reading out image data written in 
the memory 30 using the memory control circuit 22 and 
image processing circuit 20 if required (step Si 679). 
[0763] The details of the flow of this image sensing 
process step Si 679 are the same as those in the sec- 
ond embodiment that has already been described using 
Fig. 6. 

[0764] The system control circuit 50 executes a dis- 
play image process for reading out image data written in 
the memory 30 by the image sensing process in step 
S1679, and transferring the readout data to the image 
display memory 24 via the memory control circuit 22 
(stepS 1680). 

[0765] The details of the flow of this display image 
process step S1680 are the same as those in the sec- 
ond embodiment that has already been described using 
Fig. 7. 

[0766] The system control circuit 50 sets the display 
state of the image display unit 28 in a quick review dis- 
play state (step S1681), and the flow advances to step 
S1682. 

[0767] In quick review display, image data transferred 
to the image display memory 24 by the display image 
process in step S1680 is displayed on the aforemen- 
tioned image sensing region of the image display unit 
28 via the memory control circuit 22 and D/A converter 
26, thus implementing an electronic viewfinder function 
that automatically plays back a sensed image. 
[0768] The system control circuit 50 executes a com- 
pression process in which various image processes are 
done using the memory control circuit 22 and image 
processing circuit 20 if required by reading out sensed 
image data written in the memory 30, or an image is 
compressed in correspondence with the selected mode 
using the compression/expansion circuit 32 (step 
S1682). After that, the system control circuit 50 exe- 
cutes a recording process for writing image data in the 
recording medium 500 or 510 such as a memory card, 
compact flash card, or the like via the interface 90 or 94, 
and the connector 92 or 96 (step S1683). 
[0769] The details of the flow of this compression 
process step S1682 are the same as those in the sec- 
ond embodiment that has already been described using 
Fig. 8. 

[0770] When the image display unit 28 is ON, a mes- 
sage such as "BUSY" or the like that indicates that a 
write is in progress is displayed on the image display 
unit 28 during a write of image data on the recording 
medium 500 or 510. 

[0771] Furthermore, the indication unit 54 makes a 
recording medium write access indication by flickering 
an LED or the like in combination with the above mes- 



sage. 

[0772] The system control circuit 50 continues quick 
review display on the image display unit 28 until the 
shutter switch SW2 is released (step S1684). With this 
5 display the user can carefully check the sensed image 
while pressing the shutter switch SW2. 
[0773] If the shutter switch SW2 is released (step 
S1684), the system control circuit 50 sets the pano- 
ramic image sensing start flag (step S1685) to indicate 
io that the first image has already been sensed in a series 
of panoramic image sensing processes. In this manner, 
as has been described in step Si 642, the image sens- 
ing direction can be inhfoited from being reversed after 
the first image is sensed in the horizontal or vertical 
is panoramic mode. 

[0774] The system control circuit 50 sets the variable 
n to be n = n + 1 to select the next image sensing region 
(step S1686), and resets the variable mtom = 1 (step 
S1687). Then, the flow returns to step S1634 to repeat 
20 a series of panoramic image sensing processes. 

[0775] In this manner, after the control returns to step 
S1634 upon releasing the shutter switch SW2, the next 
image sensing region is selected in step S1646, the 
image sensing regions which include the previous 
25 image sensing region and display sensed images are 
set in the quick review state, and the next image sensing 
region is set in the through display state, so as to display 
the sensed images and the next image to be sensed to 
overlap each other in step S1 648, thus allowing the user 
30 to easily sense panoramic images. 

[0776] Figs. 14A and 14B that have already been 
described show examples of the image sensing window 
before and after completion of sensing of the n-th image 
in the horizontal panoramic mode. 
35 [0777] Fig. 14A shows the image sensing window 
before sensing the first image, and Fig. 14B shows the 
image sensing window upon releasing the shutter 
switch SW2 after the first image is sensed. Although no 
dedicated explanatory view is available, before the shut- 
40 ter switch SW2 is released after the first image sensing, 
the first image sensing region in Fig. 14A on the image 
sensing window is set in a quick review display state. 
[0778] Figs. 15A and 15B that have already been 
described show examples of the image sensing window 
45 before and after completion of sensing of the n-th image 
in the vertical panoramic mode. 

[0779] Fig. 15A shows the image sensing window 
before sensing the first image, and Fig. 15B shows the 
image sending window upon releasing the shutter 

so switch SW2 after the first image is sensed. Although no 
dedicated explanatory view is available, before the shut- 
ter switch SW2 is released after the first image sensing, 
the first image sensing region in Fig. 15A on the image 
sensing window is set in a quick review display state. 

55 [0780] Furthermore. Figs. 12A to 13B that have 
already been described show examples of the image 
sensing window before and after completion of sensing 
of the n-th image in the 2 x 2 panoramic mode. 
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[0781] Fig. 12A shows the image sensing window 
before the first image is sensed. Fig. 12B shows the 
image sensing window upon releasing the shutter 
switch SW2 after the first image is sensed, i.e., before 
tne second image is sensed. 5 
(0782] Fig. 13A shows the image sensing window 
upon releasing the shutter switch SW2 after the second 
image is sensed, i.e., before the third image is sensed, 
and Fig. 13B shows the image sensing window upon 
canceling the shutter switch SW2 after the third image is io 
sensed, i.e., before the fourth image is sensed. 
[0783] Although no dedicated explanatory view is 
available, before the shutter switch SW2 is released 
after the n-th image sensing, the n-th image sensing 
i egion in Figs. 1 2A to 1 3B on the image sensing window 15 
is set in a quick review display state. When the shutter 
switch SW2 is released, a through display of the (n+1)- 
th image sensing region as the next region is added. 
[0784] As described above, according to the fifth 
embodiment, since the lens unit 300 is inhibited from 20 
being exchanged by another lens unit with a different 
tocal length after the start of a series of panoramic 
image sensing processes, an image, which is unsuita- 
ble tor panoramic synthesis, can be prevented from 
being sensed and paired with the previously sensed 25 
image 

[0785] Since the current series of panoramic image 
sensing processes are temporarily interrupted and a 
new series of panoramic image sensing processes are 
started, the lens unit 300 is inhibited from being 30 
exchanged by another lens unit with a different focal 
length, and an image, which is unsuitable for panoramic 
synthesis, can be prevented from being sensed and 
paired with the previously sensed image. 
[0786] Furthermore, since various kinds of information 35 
saved in the nonvolatile memory in the lens system con- 
trol circuit 350 upon detachment of the lens unit 300 can 
be read and used when the detached lens unit 300 is 
attached to the image processing apparatus 400 again, 
the user need not repeat various setups in correspond- 40 
ence with the lens unit 300 used every time he or she 
exchanges the lens unit 300. 

[0787] The fourth and fifth embodiments of the 
present invention have been explained. 
[0788] In the description of the embodiments, if conf ir- 45 
mation otthe end of a series of panoramic image sens- 
ing processes is selected in step Si 665. the flow 
returns to step S1 631 to start a new series of panoramic 
image sensing processes. Alternatively, the flow may 
return to step S161 1 to restart setups from the pano- so 
ramie mode selection window, and then a new series of 
panoramic image sensing processes may start. 
[0789] Note that the number of types of selectable 
panoramic modes on the panoramic mode selection 
window is 3. i.e.. the 2 x 2 panoramic mode, horizontal 55 
panoramic mode, and vertical panoramic mode. How- 
ever, the present invention is not limited to these spe- 
cific modes, and one or a plurality of types of an 



arbitrary number of different panoramic modes may be 
selectable. 

[0790] Also, display of the panoramic mode selection 
window shown in Fig. 9 is not limited to this example. 
Alternatively, display with an arbitrary layout or scheme 
using an image or voice may be made. 
[0791] On the 2 x 2 panoramic mode image sensing 
window, a total of four images are two-dirnensionally 
laid out in a 2 x 2 matrix. However, the present invention 
is not limited to such matrix. For example, an arbitrary 
number of images such as a total of nine images, a total 
of 16 images, and the like may be laid out in a two- 
dimensional or three-dimensional matrix. 
[0792] On the horizontal and vertical panoramic mode 
image sensing windows, two images are continuously 
displayed. However, the present invention is not limited 
to such specific display, an arbitrary number of images 
(e.g., three images, four images, and the like) may be 
one-dimensionally cfisplayed. 

[0793] The displays of the panoramic mode image 
sensing windows shown in Figs. 1 0A to 1 5B are not lim- 
ited to these examples, and displays with arbitrary lay- 
outs and schemes using images, voices, characters, 
symbols, and the like may be made. 
[0794] In the horizontal and vertical panoramic 
modes, the image sensing direction is inhibited from 
being reversed after the first image is sensed. However, 
the present invention is not limited to this, and the image 
sensing direction may be reversed as needed. Also, the 
image sensing direction may be changed in one of arbi- 
trary directions including the right and left directions, up 
and down directions, or oblique directions. In the 
present invention, when a large number of image dis- 
play regions can be simultaneously displayed by 
increasing the display area of the image display unit 28. 
the image sensing direction can be changed in one of 
arbitrary directions including the right and left directions, 
up and down directions, or oblique directions. 
[0795] Image sensing can be redone for only the 
immediately preceding sensed image in the horizontal 
and vertical panoramic modes. However, the present 
invention is not limited to this. For example, image sens- 
ing may be redone for a sensed image an arbitrary 
number of times before. 

[0796] The next image display region for displaying an 
image to be sensed is set in the through display state, 
and that for displaying the already sensed image is set 
in the quick review display state, to display a playback 
image of the already sensed image and the next image 
to be sensed so that their boundary regions partially 
overlap each other. However, the present invention is 
not limited to such specific scheme. For example, the 
images may be displayed without making their bound- 
ary regions overlap each other. 
[0797] In the explanatory views of the panoramic 
modes in Figs. 10A to 15B, for example, comment 
words "LIVE" and "FREEZE" are added within the 
image display regions. However, the actual image dis- 
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play unit 28 may or may not display these comment 
words. 

[0798] A plurality of lens units 300 may be detachably 
attached to the image processing apparatus 400. 
[0799] The recording media 500 and 51 0 are not lim- s 
ited to memory cards such as PCMCIA cards, compact 
flash cards, or the like, hard disks, and the like, but may 
use micro DATs (digital audio tapes), magnetooptical 
disks, optical disks such as CD-Rs, CD-WRs, or the like, 
phase change optical disks such as DVDs, and the like. 10 
[0800] Also, the recording media 500 and 510 may 
use hybrid media that integrate memory cards, hard 
disks, and the like. Furthermore, such hybrid media may 
include detachable media. 

[0801 ] In the description of the fourh and fifth embod- 
iments, the recording media 500 and 510 are independ- 
ent from the image processing apparatus 400 and are 
arbitrarily connectable. One or both the recording media 
500 and 510 may be permanently connected to the 
image processing apparatus 400. 
[0802] An arbitrary number (one or a plurality) of 
image recording media 500 or 510 may be connectable 
to the image processing apparatus 400. 
[0803] In the above description, the recording media 
500 and 510 are attached to the image processing 
apparatus. However, one or a combination of a plurality 
of recording media may be used. 
[0804] Synthesis of a panoramic image has been 
described as an example in which a plurality of images 
are associated. Also, the present invention can be 
applied to a case wherein a plurality of images are 
sensed while shifting pixels, and a high-resolution 
image is obtained by synthesizing these images. 
[0805] To restate, the fourth and fifth embodiments 
can solve the conventional problem that image quality 
deteriorates considerably due to extra correction com- 
putations required to correct different focal lengths 
between the previous and current lenses exchanged 
upon generating a panoramic image by combining a 
plurality of sensed images. 

[0806] Also, the fourth and fifth embodiments can 
solve the conventional problems that the outer shape of 
a synthesized panoramic image varies in units of 
sensed image and has steps when the individual 
sensed images have considerably different angles due 
to exchange of the photographing lens with another lens 
with a different focal length. 

[0807] Furthermore, the fourth and fifth embodiments 
can solve the conventional problem that the user must 
repeat setups to obtain the previous setup state in cor- 
respondence with the photographing lens upon attach- 
ment of the detached photographing lens. 

(Another Embodiment) 

[0808] Note that the present invention may be applied 
to either a system constituted by a plurality of equip- 
ments (e.g., a host computer, an interface device, a 



reader, a printer, and the like), or an apparatus consist- 
ing of a single equipment (e.g., a copying machine, a 
facsimile apparatus, or the like). 
[0809] The objects of the present invention are also 
achieved by supplying a storage medium, which records 
a program code of a software program that can realize 
the functions of the above-mentioned embodiments to 
the system or apparatus, and reading out and executing 
the program code stored in the storage medium by a 
computer (or a CPU or MPU) of the system or appara- 
tus. 

[0810] In this case, the program code itself read out 
from the storage medium realizes the functions of the 
above-mentioned embodiments, and the storage 
medium which stores the program code constitutes the 
present invention. 

[0811] As the storage medium for supplying the pro- 
gram code, for example, a floppy disk, hard disk, optical 
disk, magneto-optical disk, CD-ROM, CD-R, magnetic 
tape, nonvolatile memory card, ROM, and the like may 
be used. 

[081 2] The functions of the above-mentioned embod- 
iments may be realized not only by executing the read- 
out program code by the computer but also by some or 
all of actual processing operations executed by an OS 
(operating system) running on the computer on the 
basis of an instruction of the program code. 
[0813] Furthermore, the functions of the above-men- 
tioned embodiments may be realized by some or all of 
actual processing operations executed by a CPU or the 
like arranged in a function extension board or a function 
extension unit, which is inserted in or connected to the 
computer, after the program code read out from the 
storage medium is written in a memory of the extension 
board or unit. 

[0814] The present invention is not limited to the 
above embodiments and various changes and modifica- 
tions may can be made within the spirit and scope of the 
present invention. Therefore, to apprise the public of the 
scope of the present invention the following claims are 
made. 

[0815] This invention has as its object to provide a 
convenient image processing apparatus which does not 
limit image sensing purposes. To achieve this object, an 
image processing apparatus has a recording medium 
for recording and playing back an image sensed by an 
image sensing unit, and an image display unit for play- 
ing back and displaying at least an image sensed before 
the current image. The image display unit has a display 
layout mode for displaying the current image sensing 
signal and at least the image sensed before the current 
image sensing signal with their partial boundary regions 
overlapping each other. 

Claims 

1. An image processing method for recording a plural- 
ity of sensed images on a recording medium, and 
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playing back and displaying the images, compris- 
ing: the image sensing step of sensing an image; 
the recording/playback step of recording and play- 
ing back the image sensed in the image sensing 
step; and the display step of playing back and dis- 
playing an image sensed at least before a current 
image, wherein the display step comprises a plural- 
ity of display layout modes for displaying the current 
image sensing signal, and a signal obtained by 
playing back the image sensed at least before the 
current image with partial boundary regions thereof 
overlapping each other. 

2. The method according to claim 1, wherein the 
image is a still image and/or a moving image. 

3. The method according to claim 1 , wherein the plu- 
rality of display layout modes of the display step 
include a first display layout mode in which the 
images are laid out in two directions, and a second 
display layout mode in which the images are laid 
out in one direction. 

4. The method according to claim 3, wherein in the 
second display layout mode, the images are laid out 
horizontally and/or vertically. 

5. An image processing apparatus for recording a plu- 
rality of sensed images on a recording medium, and 
playing back and displaying the images, compris- 
ing: image sensing means for sensing an image; 
recording/playback means for recording and play- 
ing back the image sensed by said image sensing 
means; and display means for playing back and dis- 
playing an image sensed at least before a current 
image, wherein said display means comprises a 
plurality of display layout modes for displaying the 
current image sensing signal, and a signal obtained 
by playing back the image sensed at least before 
the current image with partial boundary regions 
thereof overlapping each other. 

6. The apparatus according to claim 5, wherein the 
image is a still image and/or a moving image. 

7. The apparatus according to claim 5, wherein the 
plurality of display layout modes of said display 
means include a first display layout mode in which 
the images are laid out in two directions, and a sec- 
ond display layout mode in which the images are 
laid out in one direction. 

8. The apparatus according to claim 7. wherein in the 
second display layout mode, the images are laid out 
horizontally and/or vertically. 

9. The method according to claim 1 , further compris- 
ing a function of reversing the layout direction in the 



one direction. 

10. The apparatus according to claim 5, further com* 
prising a function of reversing the layout direction in 

5 the one direction. 

1 1 . The method according to claim 1 , wherein the dis- 
play step comprises a display layout mode for lay- 
ing out and displaying the current image sensing 

io signal, and a signal obtained by playing back the 
image sensed at least before the current image in 
two directions with partial boundary regions thereof 
overlapping each other, and includes the selection 
step of selecting an arbitrary one of display regions 

15 laid out in the display layout mode. 

12. The apparatus according to claim 5. wherein said 
display means comprises a display layout mode for 
laying out and displaying the current image sensing 

20 signal, and a signal obtained by playing back the 
image sensed at least before the current image in 
two directions with partial boundary regions thereof 
overlapping each other, and includes selection 
means for selecting an arbitrary one of display 

25 regions laid out in the display layout mode. 

1 3. A storage medium which stores a control program 
for controlling an image processing apparatus for 
recording a plurality of sensed images on a record- 

30 ing medium, and playing back and displaying the 
images, said control program having control mod- 
ules of the steps of: sensing an image; recording 
and playing back the sensed image; playing back 
and displaying an image sensed at least before a 

35 current image; and controlling to execute a plurality 
of display layout modes for displaying the current 
image sensing signal, and a signal obtained by 
playing back the image sensed at least before the 
current image with partial boundary regions thereof 

40 overlapping each other. 

14. The medium according to claim 13, wherein the 
image is a still image and/or a moving image. 

45 15. The medium according to claim 13, wherein said 
program further has a control module of the step of 
controlling to execute a function of reversing the 
layout direction in the one direction. 

so 16. The medium according to claim 13, wherein said 
control program further has a control module of the 
step of controlling to select an arbitrary one of dis- 
play regions laid out in a display layout mode for lay- 
ing out and displaying the current image sensing 

55 signal, and a signal obtained by playing back the 
image sensed at least before the current image in 
two directions with partial boundary regions thereof 
overlapping each other. 
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17. An image processing apparatus having a function 
of storing a plurality of sensed still images and/or 
moving images in storage means, comprising: 

image sensing means comprising an image 
sensing fens which can change an optical sys- 
tem condition; 

storage means for storing a plurality of images 
sensed by said image sensing means in asso- 
ciation with each other; 

optical system condition change instruction 
means for outputting an instruction for chang- 
ing the optical system condition of said image 
sensing lens; and 

control means for controlling to generate an 
alert and/or inhibit the optical system condition 
of said image sensing lens from changing upon 
reception of the instruction for changing the 
optical system condition of said image sensing 
lens from said optical system condition change 
instruction means after a first one of the plural- 
ity of images to be stored in said storage 
means in association with each other is sensed 
and stored. 

18. The apparatus according to claim 17, wherein the 
optical system condition is a focal length of said 
image sensing lens. 

19. The apparatus according to claim 17, wherein 
associating the plurality of images is obtaining a 
panoramic image by synthesizing the plurality of 
images. 

20. The apparatus according to claim 17. wherein the 
plurality of images are images sensed by perform- 
ing pixel shift, and associating the plurality of 
images is obtaining a high-resolution image by syn- 
thesizing the plurality of images sensed by perform- 
ing the pixel shift. 

21. An image processing apparatus having a function 
of storing a plurality of sensed still images and/or 
moving images in storage means, comprising: 

image sensing means comprising an image 
sensing lens which can change an optical sys- 
tem condition; 

storage means for storing a plurality of images 
sensed by said image sensing means in asso- 
ciation with each other; 

optical system condition change instruction 
means for outputting an instruction for chang- 
ing the optical system condition of said image 
sensing lens; and 

control means for controlling to start image 
sensing of a plurality of new images to be 
stored in association with each other upon 



reception of the instruction for changing the 
optical system condition of said image sensing 
lens from said optical system condition change 
instruction means after a first one of the plural - 
5 ity of images to be stored in said storage 

means in association with each other is sensed 
and stored. 

22. The apparatus according to claim 21, wherein the 
10 optical system condition is a focal length of said 

image sensing lens. 

23. The apparatus according to claim 21, wherein 
associating the plurality of images is obtaining a 

is panoramic image by synthesizing the plurality of 
images. 

24. The apparatus according to claim 21, wherein the 
plurality of images are images sensed by perform - 

20 ing pixel shift, and associating the plurality of 
images is obtaining a high-resolution image by syn- 
thesizing the plurality of images sensed by perform- 
ing the pixel shift. 

25 25. The apparatus according to claim 1 7, wherein said 
control means controls to start image sensing of a 
plurality of new images to be stored in association 
with each other after the alert is generated. 

30 26. An image processing apparatus having a function 
of storing a plurality of sensed still images and/or 
moving images in storage means, comprising: 

image sensing means comprising an image 
35 sensing lens which can change an optical sys- 

tem condition; 

storage means for storing a plurality of images 
sensed by said image sensing means in asso- 
ciation with each other; 
40 optical system condition change instruction 

means tor outputting an instruction for chang- 
ing the optical system condition of said image 
sensing lens; and 

control means for controlling to set the optical 
45 - system condition of said image sensing lens at 

an initial value before sensing of a first one of 
the plurality of images to be stored in said stor- 
age means in association with each other is 
starred. 

50 

27. The apparatus according to claim 26, wherein the 
optical system condition is a focal length of said 
image sensing lens. 

55 28. The apparatus according to claim 27, wherein the 
initial value is a focal length on a wide-angle end of 
said image sensing lens. 
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29. The apparatus according to claim 26. wherein 
associating the plurality of images is obtaining a 
panoramic image by synthesizing the plurality of 
images. 

5 

30. The apparatus according to claim 26, wherein the 
plurality of images are images sensed by perform- 
ing pixel shift, and associating the plurality of 
images is obtaining a high-resolution image by syn- 
thesizing the plurality of images sensed by perform- 10 
ing the pixel shift. 

31. A control method for an image processing appara- 
tus which comprises image sensing means com- 
prising an image sensing lens which can change an is 
optical system condition, storage means lor storing 

a plurality of images sensed by said image sensing 
means in association with each other, and optical 
system condition change instruction means for out- 
putting an instruction for changing the optical sys- 20 
tern condition of said image sensing lens, 
comprising the step of: 

generating an alert and/or inhibiting the optical 
system condition of said image sensing lens 25 
from changing upon reception of the instruction 
for changing the optical system condition of 
said image sensing lens from said optical sys- 
tem condition change instruction means after a 
first one of the plurality of images to be stored 30 
in said storage means in association with each 
other is sensed and stored. 

32. The method according to claim 31, wherein the 
optical system condition is a focal length of said 35 
image sensing lens. 

33. The method according to claim 31 , wherein associ- 
ating the plurality of images is obtaining a pano- 
ramic image by synthesizing the plurality of images. 40 

34. The method according to claim 31 , wherein the plu- 
rality of images are images sensed by performing 
pixel shift, and associating the plurality of images is 
obtaining a high-resolution image by synthesizing 45 
the plurality of images sensed by performing the 
pixel shift. 

35. A control method for an image processing appara- 
tus which comprises image sensing means com- so 
prising an image sensing lens which can change an 
optical system condition, storage means tor storing 

a plurality of images sensed by said image sensing 
means in association with each other, and optical 
system condition change instruction means for out- 55 
putting an instruction for changing the optical sys- 
tem condition of said image sensing lens, 
comprising the step of: 



starting image sensing of a plurality of new 
images to be stored in association with each 
other upon reception of the instruction for 
changing the optical system condition of said 
image sensing lens from said optical system 
condition change instruction means after a first 
one of the plurality of images to be stored in 
said storage means in association with each 
other is sensed and stored. 

36. The method according to claim 35, wherein the 
optical system condition is a focal length of said 
image sensing lens. 

37. The method according to claim 35, wherein associ- 
ating the plurality of images is obtaining a pano- 
ramic image by synthesizing the plurality of images. 

38. The method according to claim 35, wherein the plu- 
rality of images are images sensed by performing 
pixel shift, and associating the plurality of images is 
obtaining a high-resolution image by synthesizing 
the plurality of images sensed by performing the 
pixel shift. 

39. The method according to claim 31. wherein image 
sensing of a plurality of new images to be stored in 
association with each other is started after the alert 
is generated. 

40. A control method for an image processing appara- 
tus which comprises image sensing means com- 
prising an image sensing lens which can change an 
optical system condition, storage means for storing 
a plurality of images sensed by said image sensing 
means in association with each other, and optical 
system condition change instruction means for out- 
putting an instruction for changing the optical sys- 
tem condition of said image sensing lens, 
comprising the step of: 

setting the optical system condition of said 
image sensing lens at an initial value before 
sensing of a first one of the plurality of images 
to be stored in said storage means in associa- 
tion with each other is started. 

41. The method according to claim 40, wherein the 
optical system condition is a focal length of said 
image sensing lens. 

42. The method according to claim 41, wherein the ini- 
tial value is a focal length on a wide-angle end of 
said image sensing lens. 

43. The method according to claim 40, wherein associ- 
ating the plurality of images is obtaining a pano- 
ramic image by synthesizing the plurality of images. 
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44. The method according to claim 40, wherein the plu- 
rality of images are images sensed by performing 
pixel shift, and associating the plurality of images is 
obtaining a high -resolution image by synthesizing 
the plurality of images sensed by performing the 
pixel shift. 

45. A storage medium that stores a control program for 
controlling an image processing apparatus which 
comprises image sensing means comprising an 
image sensing lens which can change an optical 
system condition, storage means for storing a plu- 
rality of images sensed by said image sensing 
means in association with each other, and optical 
system condition change instruction means for out- 
putting an instruction for changing the optical sys- 
tem condition of said image sensing lens, said 
control program comprising a code of the step of: 

generating an alert and/or inhibiting the optical 
system condition of said image sensing lens 
from changing upon reception of the instruction 
for changing the optical system condition of 
said image sensing lens from said optical sys- 
tem condition change instruction means after a 
first one of the plurality of images to be stored 
in said storage means in association with each 
other is sensed and stored. 

46. The medium according to claim 45, wherein the 
optical system condition is a focal length of said 
image sensing lens. 

47. The medium according to claim 45, wherein associ- 
ating the plurality of images is obtaining a pano- 
ramic image by synthesizing the plurality of images. 

48. The medium according to claim 45. wherein the plu- 
rality of images are images sensed by performing 
pixel shift, and associating the plurality of images is 
obtaining a high-resolution image by synthesizing 

. the plurality of images sensed by performing the 
pixel shift. 

49. A storage medium that stores a control program for 
controlling an image processing apparatus which 
comprises image sensing means comprising an 
image sensing lens which can change an optical 
system condition, storage means for storing a plu- 
rality of images sensed by said image sensing 
means in association with each other, and optical 
system condition change instruction means for out- 
putting an instruction for changing the optical sys- 
tem condition of said image sensing lens, said 
control program comprising a code of the step of: 

starting image sensing of a plurality of new 
images to be stored in association with each 



other upon reception of the instruction for 
changing the optical system condition of said 
image sensing lens from said optical system 
condition change instruction means after a first 
5 one of the plurality of images to be stored in 

said storage means in association with each 
other is sensed and stored. 

50. The medium according to claim 49, wherein the 
io optical system condition is a focal length of said 

image sensing lens. 

51 . The medium according to claim 49, wherein associ- 
ating the plurality of images is obtaining a pano- 

15 ramie image by synthesizing the plurality of images. 

52. The medium according to claim 49, wherein the plu- 
rality of images are images sensed by performing 
pixel shift, and associating the plurality of images is 

20 obtaining a high-resolution image by synthesizing 
the plurality of images sensed by performing the 
pixel shift. 

53. The medium according to claim 45, wherein said 
25 control program further comprises a code of the 

step of starting image sensing of a plurality of new 
images to be stored in association with each other 
after the alert is generated. 

30 54. A storage medium that stores a control program for 
controlling an image processing apparatus which 
comprises image sensing means comprising an 
image sensing lens which can change an optical 
system condition, storage means for storing a plu- 

35 raiity of images sensed by said image sensing 
means in association with each other, and optical 
system condition change instruction means for out- 
putting an instruction for changing the optical sys- 
tem condition of said image sensing lens, said 

40 control program comprising a code of the step of: 

setting the optical system condition of said 
image sensing lens at an initial value before 
sensing of a first one of the plurality of images 
45 to be stored in said storage means in associa- 

tion with each other is started. 

55. The medium according to claim 54, wherein the 
optical system condition is a focal length of said 

so image sensing lens. 

56. The medium according to claim 55. wherein the ini- 
tial value is a focal length on a wide-angle end of 
said image sensing lens. 

55 

57. The medium according to claim 54, wherein associ- 
ating the plurality of images is obtaining a pano- 
ramic image by synthesizing the plurality of images. 
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58. The medium according to claim 54, wherein the plu- 
rality of images are images sensed by performing 
pixel shift, and associating the plurality of images is 
obtaining a high-resolution image by synthe^zrng 
the plurality of images sensed by performing the 
pixel shift. 

59. The apparatus according to any one of claims 17, 
21 and 26, wherein the change in optical system 
condition includes free attachment/detachment of 
said lens unit. 

60. An image processing apparatus having a function 
of storing a plurality of sensed still images and/or 
moving images in storage means, comprising: 

a detachable lens unit having nonvolatile stor- 
age means; 

image sensing means for sensing an object 
image formed by said lens unit; 
instruction means for instructing attach- 
ment/detachment of said lens unit; and 
control means tor controlling to permit detach- 
ment of said lens unit after information pertain- 
ing to an operation state of said lens unit and/or 
user information are/is stored in said nonvola- 
tile storage means, when said instruction 
means outputs an instruction for detaching said 
lens unit, and for reading out information per- 
taining to a use state of said lens unit and/or 
the user information stored in said nonvolatile 
storage means and re-setting an operation 
state of said image processing apparatus in 
accordance with the readout information, when 
said lens unit is attached again. 

61. The apparatus according to claim 60, wherein the 
operation state is an image sensing mode of said 
image processing apparatus. 

62. The apparatus according to claim 60. wherein the 
, operation state is a focal length setting value of a 
lens unit, a focal length of which can be changed. 

63. A control method for an image processing appara- 
tus which comprises a detachable lens unit having 
nonvolatile storage means, image sensing means 
for sensing an object image formed by said lens 
unit, and instruction means for instructing attach- 
ment/detachment of said lens unit, comprising the 
step of: 



information pertaining to a use state of said 
lens unit and/or the user information stored in 
said nonvolatile storage means and re-setting 
an operation state of said image processing 
5 apparatus in accordance with the readout infor- 

mation, when said lens unit is attached again. 

64. The method according to claim 63. wherein the 
operation state is an image sensing mode of said 

10 image processing apparatus. 

65. The method according to claim 63, wherein the 
operation state is a focal length setting value of a 
lens unit, a focal length of which can be changed. 

15 

66. A storage medium that stores a control program for 
controlling an image processing apparatus which 
comprises a detachable lens unit having nonvolatile 
storage means, image sensing means for sensing 

20 an object image formed by said lens unit, and 
instruction means for instructing attach- 
ment/detachment of said lens unit, said control pro- 
gram comprising a code of the step of: 

25 permitting detachment of said lens unit after 

information pertaining to an operation state of 
said lens unit and/or user information are/is 
stored in said nonvolatile storage means, when 
said instruction means outputs an instruction 

30 for detaching said lens unit, and reading out 

information pertaining to a use state of said 
lens unit and/or the user information stored in 
said nonvolatile storage means and re-setting 
an operation state of said image processing 

35 apparatus in accordance with the readout infor- 

mation, when said lens unit is attached again. 

67. The medium according to claim 66, wherein the 
operation state is an image sensing mode of said 

40 image processing apparatus. 

68. The medium according to claim 66, wherein the 
operation state is a focal length setting value of a 
lens unit, a focal length of which can be changed. 

45 



50 



permitting detachment of said lens unit after 
information pertaining to an operation state of 
said lens unit and/or user information are/is ss 
stored in said nonvolatile storage means, when 
said instruction means outputs an instruction 
for detaching said lens unit, and reading out 
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